2023-04-18 22:53:38,789 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827 2023-04-18 22:53:38,806 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-18 22:53:38,825 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-18 22:53:38,826 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/cluster_e7f6bc09-fd33-5a3b-ac62-21cde829d37b, deleteOnExit=true 2023-04-18 22:53:38,826 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-18 22:53:38,827 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/test.cache.data in system properties and HBase conf 2023-04-18 22:53:38,827 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/hadoop.tmp.dir in system properties and HBase conf 2023-04-18 22:53:38,828 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/hadoop.log.dir in system properties and HBase conf 2023-04-18 22:53:38,828 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-18 22:53:38,828 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-18 22:53:38,828 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-18 22:53:38,949 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-18 22:53:39,374 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-18 22:53:39,379 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-18 22:53:39,389 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-18 22:53:39,390 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-18 22:53:39,390 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-18 22:53:39,391 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-18 22:53:39,391 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-18 22:53:39,391 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-18 22:53:39,392 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-18 22:53:39,392 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-18 22:53:39,392 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/nfs.dump.dir in system properties and HBase conf 2023-04-18 22:53:39,393 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/java.io.tmpdir in system properties and HBase conf 2023-04-18 22:53:39,393 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-18 22:53:39,393 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-18 22:53:39,394 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-18 22:53:40,003 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-18 22:53:40,008 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-18 22:53:40,427 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-18 22:53:40,630 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-18 22:53:40,652 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-18 22:53:40,702 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-18 22:53:40,742 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/java.io.tmpdir/Jetty_localhost_46645_hdfs____ptelb/webapp 2023-04-18 22:53:40,893 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46645 2023-04-18 22:53:40,937 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-18 22:53:40,938 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-18 22:53:41,365 WARN [Listener at localhost/35337] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-18 22:53:41,529 WARN [Listener at localhost/35337] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-18 22:53:41,554 WARN [Listener at localhost/35337] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-18 22:53:41,563 INFO [Listener at localhost/35337] log.Slf4jLog(67): jetty-6.1.26 2023-04-18 22:53:41,569 INFO [Listener at localhost/35337] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/java.io.tmpdir/Jetty_localhost_43345_datanode____4wevek/webapp 2023-04-18 22:53:41,700 INFO [Listener at localhost/35337] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43345 2023-04-18 22:53:42,087 WARN [Listener at localhost/36651] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-18 22:53:42,713 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xcd718ce7b8d0c5e0: Processing first storage report for DS-2e174a5b-e651-466b-bd68-29b66648cc91 from datanode 0a64437a-398a-4328-a5cc-cc24e037595c 2023-04-18 22:53:42,715 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xcd718ce7b8d0c5e0: from storage DS-2e174a5b-e651-466b-bd68-29b66648cc91 node DatanodeRegistration(127.0.0.1:36779, datanodeUuid=0a64437a-398a-4328-a5cc-cc24e037595c, infoPort=46199, infoSecurePort=0, ipcPort=36651, storageInfo=lv=-57;cid=testClusterID;nsid=630837385;c=1681858420082), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-18 22:53:42,715 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xcd718ce7b8d0c5e0: Processing first storage report for DS-cb1e8579-8e77-4833-b5ee-0e5379dcb47c from datanode 0a64437a-398a-4328-a5cc-cc24e037595c 2023-04-18 22:53:42,715 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xcd718ce7b8d0c5e0: from storage DS-cb1e8579-8e77-4833-b5ee-0e5379dcb47c node DatanodeRegistration(127.0.0.1:36779, datanodeUuid=0a64437a-398a-4328-a5cc-cc24e037595c, infoPort=46199, infoSecurePort=0, ipcPort=36651, storageInfo=lv=-57;cid=testClusterID;nsid=630837385;c=1681858420082), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-18 22:53:42,759 DEBUG [Listener at localhost/36651] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827 2023-04-18 22:53:42,829 INFO [Listener at localhost/36651] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/cluster_e7f6bc09-fd33-5a3b-ac62-21cde829d37b/zookeeper_0, clientPort=50825, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/cluster_e7f6bc09-fd33-5a3b-ac62-21cde829d37b/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/cluster_e7f6bc09-fd33-5a3b-ac62-21cde829d37b/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-18 22:53:42,844 INFO [Listener at localhost/36651] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=50825 2023-04-18 22:53:42,854 INFO [Listener at localhost/36651] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-18 22:53:42,856 INFO [Listener at localhost/36651] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-18 22:53:43,521 INFO [Listener at localhost/36651] util.FSUtils(471): Created version file at hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0 with version=8 2023-04-18 22:53:43,521 INFO [Listener at localhost/36651] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/hbase-staging 2023-04-18 22:53:43,825 INFO [Listener at localhost/36651] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-18 22:53:44,289 INFO [Listener at localhost/36651] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-18 22:53:44,322 INFO [Listener at localhost/36651] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-18 22:53:44,323 INFO [Listener at localhost/36651] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-18 22:53:44,323 INFO [Listener at localhost/36651] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-18 22:53:44,323 INFO [Listener at localhost/36651] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-18 22:53:44,323 INFO [Listener at localhost/36651] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-18 22:53:44,472 INFO [Listener at localhost/36651] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-18 22:53:44,551 DEBUG [Listener at localhost/36651] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-18 22:53:44,655 INFO [Listener at localhost/36651] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46739 2023-04-18 22:53:44,666 INFO [Listener at localhost/36651] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-18 22:53:44,668 INFO [Listener at localhost/36651] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-18 22:53:44,689 INFO [Listener at localhost/36651] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46739 connecting to ZooKeeper ensemble=127.0.0.1:50825 2023-04-18 22:53:44,730 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:467390x0, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-18 22:53:44,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:46739-0x101997ad1e40000 connected 2023-04-18 22:53:44,772 DEBUG [Listener at localhost/36651] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-18 22:53:44,773 DEBUG [Listener at localhost/36651] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-18 22:53:44,777 DEBUG [Listener at localhost/36651] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-18 22:53:44,785 DEBUG [Listener at localhost/36651] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46739 2023-04-18 22:53:44,785 DEBUG [Listener at localhost/36651] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46739 2023-04-18 22:53:44,786 DEBUG [Listener at localhost/36651] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46739 2023-04-18 22:53:44,786 DEBUG [Listener at localhost/36651] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46739 2023-04-18 22:53:44,788 DEBUG [Listener at localhost/36651] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46739 2023-04-18 22:53:44,794 INFO [Listener at localhost/36651] master.HMaster(444): hbase.rootdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0, hbase.cluster.distributed=false 2023-04-18 22:53:44,870 INFO [Listener at localhost/36651] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-18 22:53:44,870 INFO [Listener at localhost/36651] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-18 22:53:44,870 INFO [Listener at localhost/36651] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-18 22:53:44,870 INFO [Listener at localhost/36651] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-18 22:53:44,870 INFO [Listener at localhost/36651] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-18 22:53:44,870 INFO [Listener at localhost/36651] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-18 22:53:44,878 INFO [Listener at localhost/36651] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-18 22:53:44,882 INFO [Listener at localhost/36651] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35971 2023-04-18 22:53:44,885 INFO [Listener at localhost/36651] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-18 22:53:44,894 DEBUG [Listener at localhost/36651] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-18 22:53:44,895 INFO [Listener at localhost/36651] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-18 22:53:44,898 INFO [Listener at localhost/36651] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-18 22:53:44,899 INFO [Listener at localhost/36651] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35971 connecting to ZooKeeper ensemble=127.0.0.1:50825 2023-04-18 22:53:44,902 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:359710x0, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-18 22:53:44,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:35971-0x101997ad1e40001 connected 2023-04-18 22:53:44,904 DEBUG [Listener at localhost/36651] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-18 22:53:44,905 DEBUG [Listener at localhost/36651] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-18 22:53:44,906 DEBUG [Listener at localhost/36651] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-18 22:53:44,906 DEBUG [Listener at localhost/36651] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35971 2023-04-18 22:53:44,907 DEBUG [Listener at localhost/36651] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35971 2023-04-18 22:53:44,907 DEBUG [Listener at localhost/36651] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35971 2023-04-18 22:53:44,907 DEBUG [Listener at localhost/36651] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35971 2023-04-18 22:53:44,908 DEBUG [Listener at localhost/36651] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35971 2023-04-18 22:53:44,910 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,46739,1681858423665 2023-04-18 22:53:44,919 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-18 22:53:44,921 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,46739,1681858423665 2023-04-18 22:53:44,940 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-18 22:53:44,940 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-18 22:53:44,940 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 22:53:44,941 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-18 22:53:44,943 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,46739,1681858423665 from backup master directory 2023-04-18 22:53:44,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-18 22:53:44,946 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,46739,1681858423665 2023-04-18 22:53:44,946 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-18 22:53:44,947 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-18 22:53:44,947 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,46739,1681858423665 2023-04-18 22:53:44,951 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-18 22:53:44,953 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-18 22:53:45,046 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/hbase.id with ID: b0194321-9851-4ac5-bfe7-6bb494eb118f 2023-04-18 22:53:45,090 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-18 22:53:45,105 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 22:53:45,153 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x1414fc6e to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:53:45,189 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@33d86134, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:53:45,213 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 22:53:45,215 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-18 22:53:45,232 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-18 22:53:45,233 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-18 22:53:45,234 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:53:45,239 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:53:45,240 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-18 22:53:45,275 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/MasterData/data/master/store-tmp 2023-04-18 22:53:45,304 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:53:45,305 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-18 22:53:45,305 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-18 22:53:45,305 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-18 22:53:45,305 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-18 22:53:45,305 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-18 22:53:45,305 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-18 22:53:45,305 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-18 22:53:45,307 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/MasterData/WALs/jenkins-hbase4.apache.org,46739,1681858423665 2023-04-18 22:53:45,336 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46739%2C1681858423665, suffix=, logDir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/MasterData/WALs/jenkins-hbase4.apache.org,46739,1681858423665, archiveDir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/MasterData/oldWALs, maxLogs=10 2023-04-18 22:53:45,400 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36779,DS-2e174a5b-e651-466b-bd68-29b66648cc91,DISK] 2023-04-18 22:53:45,409 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:53:45,482 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/MasterData/WALs/jenkins-hbase4.apache.org,46739,1681858423665/jenkins-hbase4.apache.org%2C46739%2C1681858423665.1681858425351 2023-04-18 22:53:45,482 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36779,DS-2e174a5b-e651-466b-bd68-29b66648cc91,DISK]] 2023-04-18 22:53:45,483 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-18 22:53:45,483 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:53:45,487 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-18 22:53:45,488 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-18 22:53:45,550 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-18 22:53:45,558 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-18 22:53:45,585 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-18 22:53:45,599 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:53:45,605 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-18 22:53:45,607 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-18 22:53:45,628 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-18 22:53:45,633 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 22:53:45,635 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67234086, jitterRate=0.0018659532070159912}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-18 22:53:45,635 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-18 22:53:45,637 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-18 22:53:45,660 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-18 22:53:45,660 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-18 22:53:45,663 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-18 22:53:45,665 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-18 22:53:45,712 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 46 msec 2023-04-18 22:53:45,712 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-18 22:53:45,740 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-18 22:53:45,746 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-18 22:53:45,774 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-18 22:53:45,779 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-18 22:53:45,781 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-18 22:53:45,786 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-18 22:53:45,791 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-18 22:53:45,800 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 22:53:45,801 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-18 22:53:45,802 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-18 22:53:45,819 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-18 22:53:45,825 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-18 22:53:45,825 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-18 22:53:45,825 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 22:53:45,826 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,46739,1681858423665, sessionid=0x101997ad1e40000, setting cluster-up flag (Was=false) 2023-04-18 22:53:45,841 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 22:53:45,848 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-18 22:53:45,849 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46739,1681858423665 2023-04-18 22:53:45,855 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 22:53:45,860 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-18 22:53:45,862 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46739,1681858423665 2023-04-18 22:53:45,865 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.hbase-snapshot/.tmp 2023-04-18 22:53:45,912 INFO [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(952): ClusterId : b0194321-9851-4ac5-bfe7-6bb494eb118f 2023-04-18 22:53:45,917 DEBUG [RS:0;jenkins-hbase4:35971] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-18 22:53:45,925 DEBUG [RS:0;jenkins-hbase4:35971] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-18 22:53:45,925 DEBUG [RS:0;jenkins-hbase4:35971] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-18 22:53:45,929 DEBUG [RS:0;jenkins-hbase4:35971] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-18 22:53:45,930 DEBUG [RS:0;jenkins-hbase4:35971] zookeeper.ReadOnlyZKClient(139): Connect 0x4ed325d2 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:53:45,935 DEBUG [RS:0;jenkins-hbase4:35971] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4014d278, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:53:45,936 DEBUG [RS:0;jenkins-hbase4:35971] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@442b4ea2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-18 22:53:45,969 DEBUG [RS:0;jenkins-hbase4:35971] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:35971 2023-04-18 22:53:45,974 INFO [RS:0;jenkins-hbase4:35971] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-18 22:53:45,974 INFO [RS:0;jenkins-hbase4:35971] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-18 22:53:45,974 DEBUG [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1023): About to register with Master. 2023-04-18 22:53:45,977 INFO [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46739,1681858423665 with isa=jenkins-hbase4.apache.org/172.31.14.131:35971, startcode=1681858424869 2023-04-18 22:53:45,995 DEBUG [RS:0;jenkins-hbase4:35971] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-18 22:53:45,996 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-18 22:53:46,008 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-18 22:53:46,008 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-18 22:53:46,008 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-18 22:53:46,008 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-18 22:53:46,008 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-18 22:53:46,009 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 22:53:46,009 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-18 22:53:46,009 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 22:53:46,012 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681858456012 2023-04-18 22:53:46,014 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-18 22:53:46,024 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-18 22:53:46,024 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-18 22:53:46,028 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-18 22:53:46,035 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-18 22:53:46,036 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-18 22:53:46,037 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-18 22:53:46,037 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-18 22:53:46,037 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-18 22:53:46,038 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-18 22:53:46,045 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-18 22:53:46,047 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-18 22:53:46,048 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-18 22:53:46,052 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-18 22:53:46,053 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-18 22:53:46,057 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681858426055,5,FailOnTimeoutGroup] 2023-04-18 22:53:46,064 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681858426057,5,FailOnTimeoutGroup] 2023-04-18 22:53:46,064 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-18 22:53:46,064 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-18 22:53:46,066 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-18 22:53:46,066 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-18 22:53:46,072 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35407, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-18 22:53:46,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:46,092 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-18 22:53:46,094 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-18 22:53:46,094 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0 2023-04-18 22:53:46,113 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:53:46,114 DEBUG [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0 2023-04-18 22:53:46,114 DEBUG [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:35337 2023-04-18 22:53:46,114 DEBUG [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-18 22:53:46,121 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-18 22:53:46,122 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-18 22:53:46,123 DEBUG [RS:0;jenkins-hbase4:35971] zookeeper.ZKUtil(162): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:46,124 WARN [RS:0;jenkins-hbase4:35971] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-18 22:53:46,124 INFO [RS:0;jenkins-hbase4:35971] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-18 22:53:46,125 DEBUG [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1947): logDir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/WALs/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:46,125 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,35971,1681858424869] 2023-04-18 22:53:46,126 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/info 2023-04-18 22:53:46,128 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-18 22:53:46,130 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:53:46,130 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-18 22:53:46,133 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/rep_barrier 2023-04-18 22:53:46,134 DEBUG [RS:0;jenkins-hbase4:35971] zookeeper.ZKUtil(162): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:46,135 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-18 22:53:46,136 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:53:46,136 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-18 22:53:46,139 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/table 2023-04-18 22:53:46,140 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-18 22:53:46,141 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:53:46,143 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740 2023-04-18 22:53:46,144 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740 2023-04-18 22:53:46,147 DEBUG [RS:0;jenkins-hbase4:35971] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-18 22:53:46,148 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 22:53:46,151 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-18 22:53:46,155 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 22:53:46,157 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59488242, jitterRate=-0.11355611681938171}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 22:53:46,157 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-18 22:53:46,157 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-18 22:53:46,157 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-18 22:53:46,157 INFO [RS:0;jenkins-hbase4:35971] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-18 22:53:46,157 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-18 22:53:46,157 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-18 22:53:46,157 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-18 22:53:46,164 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-18 22:53:46,164 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-18 22:53:46,170 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-18 22:53:46,170 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-18 22:53:46,180 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-18 22:53:46,180 INFO [RS:0;jenkins-hbase4:35971] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-18 22:53:46,191 INFO [RS:0;jenkins-hbase4:35971] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-18 22:53:46,191 INFO [RS:0;jenkins-hbase4:35971] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-18 22:53:46,192 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-18 22:53:46,192 INFO [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-18 22:53:46,195 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-18 22:53:46,201 INFO [RS:0;jenkins-hbase4:35971] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-18 22:53:46,202 DEBUG [RS:0;jenkins-hbase4:35971] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 22:53:46,202 DEBUG [RS:0;jenkins-hbase4:35971] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 22:53:46,202 DEBUG [RS:0;jenkins-hbase4:35971] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 22:53:46,203 DEBUG [RS:0;jenkins-hbase4:35971] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 22:53:46,203 DEBUG [RS:0;jenkins-hbase4:35971] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 22:53:46,203 DEBUG [RS:0;jenkins-hbase4:35971] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-18 22:53:46,203 DEBUG [RS:0;jenkins-hbase4:35971] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 22:53:46,203 DEBUG [RS:0;jenkins-hbase4:35971] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 22:53:46,203 DEBUG [RS:0;jenkins-hbase4:35971] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 22:53:46,204 DEBUG [RS:0;jenkins-hbase4:35971] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-18 22:53:46,211 INFO [RS:0;jenkins-hbase4:35971] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-18 22:53:46,211 INFO [RS:0;jenkins-hbase4:35971] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-18 22:53:46,211 INFO [RS:0;jenkins-hbase4:35971] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-18 22:53:46,227 INFO [RS:0;jenkins-hbase4:35971] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-18 22:53:46,230 INFO [RS:0;jenkins-hbase4:35971] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35971,1681858424869-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-18 22:53:46,249 INFO [RS:0;jenkins-hbase4:35971] regionserver.Replication(203): jenkins-hbase4.apache.org,35971,1681858424869 started 2023-04-18 22:53:46,250 INFO [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,35971,1681858424869, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:35971, sessionid=0x101997ad1e40001 2023-04-18 22:53:46,250 DEBUG [RS:0;jenkins-hbase4:35971] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-18 22:53:46,250 DEBUG [RS:0;jenkins-hbase4:35971] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:46,250 DEBUG [RS:0;jenkins-hbase4:35971] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35971,1681858424869' 2023-04-18 22:53:46,250 DEBUG [RS:0;jenkins-hbase4:35971] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:46,251 DEBUG [RS:0;jenkins-hbase4:35971] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:46,252 DEBUG [RS:0;jenkins-hbase4:35971] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-18 22:53:46,252 DEBUG [RS:0;jenkins-hbase4:35971] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-18 22:53:46,252 DEBUG [RS:0;jenkins-hbase4:35971] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:46,252 DEBUG [RS:0;jenkins-hbase4:35971] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35971,1681858424869' 2023-04-18 22:53:46,252 DEBUG [RS:0;jenkins-hbase4:35971] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-18 22:53:46,252 DEBUG [RS:0;jenkins-hbase4:35971] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-18 22:53:46,253 DEBUG [RS:0;jenkins-hbase4:35971] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-18 22:53:46,253 INFO [RS:0;jenkins-hbase4:35971] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-18 22:53:46,253 INFO [RS:0;jenkins-hbase4:35971] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-18 22:53:46,348 DEBUG [jenkins-hbase4:46739] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-18 22:53:46,350 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35971,1681858424869, state=OPENING 2023-04-18 22:53:46,359 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-18 22:53:46,360 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 22:53:46,361 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-18 22:53:46,365 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:53:46,366 INFO [RS:0;jenkins-hbase4:35971] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35971%2C1681858424869, suffix=, logDir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/WALs/jenkins-hbase4.apache.org,35971,1681858424869, archiveDir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/oldWALs, maxLogs=32 2023-04-18 22:53:46,391 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36779,DS-2e174a5b-e651-466b-bd68-29b66648cc91,DISK] 2023-04-18 22:53:46,394 INFO [RS:0;jenkins-hbase4:35971] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/WALs/jenkins-hbase4.apache.org,35971,1681858424869/jenkins-hbase4.apache.org%2C35971%2C1681858424869.1681858426370 2023-04-18 22:53:46,396 DEBUG [RS:0;jenkins-hbase4:35971] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36779,DS-2e174a5b-e651-466b-bd68-29b66648cc91,DISK]] 2023-04-18 22:53:46,559 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:46,562 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-18 22:53:46,567 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45974, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-18 22:53:46,582 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-18 22:53:46,582 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-18 22:53:46,587 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35971%2C1681858424869.meta, suffix=.meta, logDir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/WALs/jenkins-hbase4.apache.org,35971,1681858424869, archiveDir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/oldWALs, maxLogs=32 2023-04-18 22:53:46,610 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36779,DS-2e174a5b-e651-466b-bd68-29b66648cc91,DISK] 2023-04-18 22:53:46,625 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/WALs/jenkins-hbase4.apache.org,35971,1681858424869/jenkins-hbase4.apache.org%2C35971%2C1681858424869.meta.1681858426589.meta 2023-04-18 22:53:46,626 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36779,DS-2e174a5b-e651-466b-bd68-29b66648cc91,DISK]] 2023-04-18 22:53:46,627 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-18 22:53:46,629 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-18 22:53:46,650 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-18 22:53:46,656 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-18 22:53:46,663 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-18 22:53:46,663 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:53:46,663 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-18 22:53:46,663 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-18 22:53:46,667 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-18 22:53:46,669 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/info 2023-04-18 22:53:46,669 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/info 2023-04-18 22:53:46,670 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-18 22:53:46,671 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:53:46,672 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-18 22:53:46,681 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/rep_barrier 2023-04-18 22:53:46,681 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/rep_barrier 2023-04-18 22:53:46,682 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-18 22:53:46,684 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:53:46,684 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-18 22:53:46,686 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/table 2023-04-18 22:53:46,686 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/table 2023-04-18 22:53:46,687 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-18 22:53:46,688 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:53:46,690 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740 2023-04-18 22:53:46,694 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740 2023-04-18 22:53:46,699 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 22:53:46,702 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-18 22:53:46,704 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69671607, jitterRate=0.038187846541404724}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 22:53:46,704 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-18 22:53:46,715 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681858426548 2023-04-18 22:53:46,734 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-18 22:53:46,735 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-18 22:53:46,736 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35971,1681858424869, state=OPEN 2023-04-18 22:53:46,739 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-18 22:53:46,739 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-18 22:53:46,744 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-18 22:53:46,744 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35971,1681858424869 in 374 msec 2023-04-18 22:53:46,750 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-18 22:53:46,751 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 567 msec 2023-04-18 22:53:46,756 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 840 msec 2023-04-18 22:53:46,756 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681858426756, completionTime=-1 2023-04-18 22:53:46,756 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-18 22:53:46,757 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-18 22:53:46,833 DEBUG [hconnection-0x18afb399-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:53:46,839 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45986, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:53:46,862 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-18 22:53:46,862 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681858486862 2023-04-18 22:53:46,862 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681858546862 2023-04-18 22:53:46,862 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 105 msec 2023-04-18 22:53:46,888 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46739,1681858423665-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-18 22:53:46,888 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46739,1681858423665-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-18 22:53:46,889 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46739,1681858423665-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-18 22:53:46,891 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:46739, period=300000, unit=MILLISECONDS is enabled. 2023-04-18 22:53:46,891 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-18 22:53:46,898 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-18 22:53:46,907 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-18 22:53:46,909 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-18 22:53:46,921 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-18 22:53:46,924 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-18 22:53:46,928 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-18 22:53:46,957 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/hbase/namespace/f5ed6cadc5eb44dbe5342fc8a2527565 2023-04-18 22:53:46,962 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/hbase/namespace/f5ed6cadc5eb44dbe5342fc8a2527565 empty. 2023-04-18 22:53:46,963 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/hbase/namespace/f5ed6cadc5eb44dbe5342fc8a2527565 2023-04-18 22:53:46,963 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-18 22:53:46,992 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-18 22:53:46,997 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => f5ed6cadc5eb44dbe5342fc8a2527565, NAME => 'hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp 2023-04-18 22:53:47,011 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:53:47,011 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing f5ed6cadc5eb44dbe5342fc8a2527565, disabling compactions & flushes 2023-04-18 22:53:47,011 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565. 2023-04-18 22:53:47,011 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565. 2023-04-18 22:53:47,011 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565. after waiting 0 ms 2023-04-18 22:53:47,011 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565. 2023-04-18 22:53:47,011 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565. 2023-04-18 22:53:47,011 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for f5ed6cadc5eb44dbe5342fc8a2527565: 2023-04-18 22:53:47,017 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-18 22:53:47,033 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681858427021"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681858427021"}]},"ts":"1681858427021"} 2023-04-18 22:53:47,066 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-18 22:53:47,068 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-18 22:53:47,072 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858427068"}]},"ts":"1681858427068"} 2023-04-18 22:53:47,078 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-18 22:53:47,088 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=f5ed6cadc5eb44dbe5342fc8a2527565, ASSIGN}] 2023-04-18 22:53:47,091 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=f5ed6cadc5eb44dbe5342fc8a2527565, ASSIGN 2023-04-18 22:53:47,093 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=f5ed6cadc5eb44dbe5342fc8a2527565, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35971,1681858424869; forceNewPlan=false, retain=false 2023-04-18 22:53:47,245 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=f5ed6cadc5eb44dbe5342fc8a2527565, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:47,245 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681858427244"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858427244"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858427244"}]},"ts":"1681858427244"} 2023-04-18 22:53:47,250 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure f5ed6cadc5eb44dbe5342fc8a2527565, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:53:47,412 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565. 2023-04-18 22:53:47,413 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f5ed6cadc5eb44dbe5342fc8a2527565, NAME => 'hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565.', STARTKEY => '', ENDKEY => ''} 2023-04-18 22:53:47,414 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace f5ed6cadc5eb44dbe5342fc8a2527565 2023-04-18 22:53:47,414 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:53:47,415 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f5ed6cadc5eb44dbe5342fc8a2527565 2023-04-18 22:53:47,415 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f5ed6cadc5eb44dbe5342fc8a2527565 2023-04-18 22:53:47,420 INFO [StoreOpener-f5ed6cadc5eb44dbe5342fc8a2527565-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region f5ed6cadc5eb44dbe5342fc8a2527565 2023-04-18 22:53:47,423 DEBUG [StoreOpener-f5ed6cadc5eb44dbe5342fc8a2527565-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/namespace/f5ed6cadc5eb44dbe5342fc8a2527565/info 2023-04-18 22:53:47,423 DEBUG [StoreOpener-f5ed6cadc5eb44dbe5342fc8a2527565-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/namespace/f5ed6cadc5eb44dbe5342fc8a2527565/info 2023-04-18 22:53:47,424 INFO [StoreOpener-f5ed6cadc5eb44dbe5342fc8a2527565-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f5ed6cadc5eb44dbe5342fc8a2527565 columnFamilyName info 2023-04-18 22:53:47,425 INFO [StoreOpener-f5ed6cadc5eb44dbe5342fc8a2527565-1] regionserver.HStore(310): Store=f5ed6cadc5eb44dbe5342fc8a2527565/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:53:47,427 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/namespace/f5ed6cadc5eb44dbe5342fc8a2527565 2023-04-18 22:53:47,428 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/namespace/f5ed6cadc5eb44dbe5342fc8a2527565 2023-04-18 22:53:47,433 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f5ed6cadc5eb44dbe5342fc8a2527565 2023-04-18 22:53:47,438 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/namespace/f5ed6cadc5eb44dbe5342fc8a2527565/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 22:53:47,438 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f5ed6cadc5eb44dbe5342fc8a2527565; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69059397, jitterRate=0.02906520664691925}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-18 22:53:47,439 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f5ed6cadc5eb44dbe5342fc8a2527565: 2023-04-18 22:53:47,441 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565., pid=6, masterSystemTime=1681858427405 2023-04-18 22:53:47,446 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565. 2023-04-18 22:53:47,446 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565. 2023-04-18 22:53:47,448 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=f5ed6cadc5eb44dbe5342fc8a2527565, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:47,449 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681858427447"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681858427447"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681858427447"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681858427447"}]},"ts":"1681858427447"} 2023-04-18 22:53:47,458 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-18 22:53:47,458 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure f5ed6cadc5eb44dbe5342fc8a2527565, server=jenkins-hbase4.apache.org,35971,1681858424869 in 204 msec 2023-04-18 22:53:47,462 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-18 22:53:47,463 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=f5ed6cadc5eb44dbe5342fc8a2527565, ASSIGN in 371 msec 2023-04-18 22:53:47,464 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-18 22:53:47,465 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858427464"}]},"ts":"1681858427464"} 2023-04-18 22:53:47,468 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-18 22:53:47,473 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-18 22:53:47,477 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 564 msec 2023-04-18 22:53:47,523 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-18 22:53:47,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-18 22:53:47,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 22:53:47,566 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-18 22:53:47,586 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-18 22:53:47,595 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 38 msec 2023-04-18 22:53:47,601 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-18 22:53:47,617 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-18 22:53:47,624 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 21 msec 2023-04-18 22:53:47,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-18 22:53:47,646 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-18 22:53:47,646 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.698sec 2023-04-18 22:53:47,648 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-18 22:53:47,650 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-18 22:53:47,650 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-18 22:53:47,651 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46739,1681858423665-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-18 22:53:47,652 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46739,1681858423665-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-18 22:53:47,664 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-18 22:53:47,719 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x51f71277 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:53:47,725 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ed65498, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:53:47,741 DEBUG [hconnection-0x35d25957-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:53:47,753 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45990, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:53:47,764 INFO [Listener at localhost/36651] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,46739,1681858423665 2023-04-18 22:53:47,786 INFO [Listener at localhost/36651] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=205, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=366, ProcessCount=169, AvailableMemoryMB=7202 2023-04-18 22:53:47,803 DEBUG [Listener at localhost/36651] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-18 22:53:47,807 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54076, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-18 22:53:47,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 22:53:47,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 22:53:47,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-18 22:53:47,826 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-18 22:53:47,828 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-18 22:53:47,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-18 22:53:47,835 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:47,836 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36 empty. 2023-04-18 22:53:47,837 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:47,838 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 22:53:47,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-18 22:53:47,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-18 22:53:48,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-18 22:53:48,256 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 22:53:48,258 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3d5ba4e7a99f0fcc1148990ad0a52e36, NAME => 'TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp 2023-04-18 22:53:48,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-18 22:53:48,672 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:53:48,673 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 3d5ba4e7a99f0fcc1148990ad0a52e36, disabling compactions & flushes 2023-04-18 22:53:48,673 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:48,673 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:48,673 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. after waiting 0 ms 2023-04-18 22:53:48,673 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:48,673 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:48,673 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:48,678 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-18 22:53:48,680 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858428679"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681858428679"}]},"ts":"1681858428679"} 2023-04-18 22:53:48,682 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-18 22:53:48,684 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-18 22:53:48,684 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858428684"}]},"ts":"1681858428684"} 2023-04-18 22:53:48,686 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-18 22:53:48,690 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3d5ba4e7a99f0fcc1148990ad0a52e36, ASSIGN}] 2023-04-18 22:53:48,692 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3d5ba4e7a99f0fcc1148990ad0a52e36, ASSIGN 2023-04-18 22:53:48,694 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3d5ba4e7a99f0fcc1148990ad0a52e36, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35971,1681858424869; forceNewPlan=false, retain=false 2023-04-18 22:53:48,845 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=3d5ba4e7a99f0fcc1148990ad0a52e36, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:48,846 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858428845"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858428845"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858428845"}]},"ts":"1681858428845"} 2023-04-18 22:53:48,850 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:53:48,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-18 22:53:49,010 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:49,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3d5ba4e7a99f0fcc1148990ad0a52e36, NAME => 'TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.', STARTKEY => '', ENDKEY => ''} 2023-04-18 22:53:49,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:49,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:53:49,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:49,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:49,013 INFO [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:49,015 DEBUG [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A 2023-04-18 22:53:49,015 DEBUG [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A 2023-04-18 22:53:49,017 INFO [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:53:49,017 INFO [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3d5ba4e7a99f0fcc1148990ad0a52e36 columnFamilyName A 2023-04-18 22:53:49,018 INFO [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] regionserver.HStore(310): Store=3d5ba4e7a99f0fcc1148990ad0a52e36/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:53:49,018 INFO [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:49,021 DEBUG [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B 2023-04-18 22:53:49,021 DEBUG [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B 2023-04-18 22:53:49,021 INFO [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:53:49,021 INFO [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3d5ba4e7a99f0fcc1148990ad0a52e36 columnFamilyName B 2023-04-18 22:53:49,022 INFO [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] regionserver.HStore(310): Store=3d5ba4e7a99f0fcc1148990ad0a52e36/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:53:49,022 INFO [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:49,025 DEBUG [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C 2023-04-18 22:53:49,025 DEBUG [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C 2023-04-18 22:53:49,025 INFO [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:53:49,025 INFO [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3d5ba4e7a99f0fcc1148990ad0a52e36 columnFamilyName C 2023-04-18 22:53:49,026 INFO [StoreOpener-3d5ba4e7a99f0fcc1148990ad0a52e36-1] regionserver.HStore(310): Store=3d5ba4e7a99f0fcc1148990ad0a52e36/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:53:49,027 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:49,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:49,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:49,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 22:53:49,070 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:49,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 22:53:49,075 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 3d5ba4e7a99f0fcc1148990ad0a52e36; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71053092, jitterRate=0.058773577213287354}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 22:53:49,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:49,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., pid=11, masterSystemTime=1681858429003 2023-04-18 22:53:49,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:49,079 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:49,081 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=3d5ba4e7a99f0fcc1148990ad0a52e36, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:49,081 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858429080"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681858429080"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681858429080"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681858429080"}]},"ts":"1681858429080"} 2023-04-18 22:53:49,087 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-18 22:53:49,087 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 in 235 msec 2023-04-18 22:53:49,090 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-18 22:53:49,091 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3d5ba4e7a99f0fcc1148990ad0a52e36, ASSIGN in 397 msec 2023-04-18 22:53:49,092 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-18 22:53:49,092 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858429092"}]},"ts":"1681858429092"} 2023-04-18 22:53:49,094 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-18 22:53:49,097 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-18 22:53:49,100 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2760 sec 2023-04-18 22:53:49,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-18 22:53:49,964 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-18 22:53:49,969 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x7d9fc901 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:53:49,984 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44ef43e3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:53:49,987 DEBUG [Listener at localhost/36651] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:53:49,991 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46002, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:53:49,994 DEBUG [Listener at localhost/36651] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-18 22:53:49,997 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54080, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-18 22:53:50,005 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x79eb61dd to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:53:50,011 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61e72c30, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:53:50,012 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x748f869b to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:53:50,016 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e16d41b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:53:50,017 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x5e91d243 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:53:50,021 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fea27db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:53:50,021 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x30eb7fb2 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:53:50,026 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e728705, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:53:50,026 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x46192d17 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:53:50,031 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63859c62, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:53:50,032 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x60b8c4c7 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:53:50,036 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49a45b6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:53:50,036 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x49d71120 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:53:50,076 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fef7a21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:53:50,077 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x2879e2e1 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:53:50,081 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cd3abbd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:53:50,082 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x0d24bbb7 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:53:50,091 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6aad2e18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:53:50,100 DEBUG [hconnection-0x57af7f7a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:53:50,101 DEBUG [hconnection-0x5b12d690-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:53:50,106 DEBUG [hconnection-0x1da1ae0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:53:50,107 DEBUG [hconnection-0x146369fb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:53:50,108 DEBUG [hconnection-0x2d3cf6e2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:53:50,109 DEBUG [hconnection-0x5df5bda7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:53:50,110 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46024, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:53:50,112 DEBUG [hconnection-0x1bb9bf9e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:53:50,112 DEBUG [hconnection-0x459180a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:53:50,116 DEBUG [hconnection-0x44866d33-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:53:50,116 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46014, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:53:50,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:50,124 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46036, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:53:50,128 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46030, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:53:50,133 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46044, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:53:50,136 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46054, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:53:50,138 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46068, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:53:50,144 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46074, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:53:50,146 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46062, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:53:50,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:50,163 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:50,164 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:50,165 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:50,165 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:50,166 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:50,166 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:50,170 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:50,170 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:50,176 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:50,176 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:50,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:50,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:50,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:50,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:50,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:50,194 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:50,194 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:50,196 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:50,197 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:50,197 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:50,203 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:50,198 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:50,214 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-18 22:53:50,228 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:53:50,229 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:50,229 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:53:50,230 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:50,230 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:53:50,230 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:50,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:50,247 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:50,375 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/a7b990a2771f4953ae61cd7f43f5ff31 2023-04-18 22:53:50,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:50,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858490356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:50,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858490363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858430322, max timestamp=1681858430362Num uniques 859; 2023-04-18 22:53:50,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:53:50,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681858430322, max timestamp=1681858430369Num uniques 859; 2023-04-18 22:53:50,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-18 22:53:50,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681858430322, max timestamp=1681858430362Num uniques 859; 2023-04-18 22:53:50,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-18 22:53:50,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681858430426, max timestamp=1681858430465Num uniques 1018; 2023-04-18 22:53:50,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-18 22:53:50,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681858430426, max timestamp=1681858430465Num uniques 1009; 2023-04-18 22:53:50,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-18 22:53:50,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681858430426, max timestamp=1681858430465Num uniques 1009; 2023-04-18 22:53:50,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-18 22:53:50,550 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/768745411809421f9ccadbe76b1b29d6 2023-04-18 22:53:50,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:50,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858490564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:50,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858490572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681858430551, max timestamp=1681858430570Num uniques 1077; 2023-04-18 22:53:50,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-18 22:53:50,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681858430551, max timestamp=1681858430573Num uniques 1009; 2023-04-18 22:53:50,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-18 22:53:50,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681858430551, max timestamp=1681858430570Num uniques 1009; 2023-04-18 22:53:50,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-18 22:53:50,668 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/fcc98a05222a4ce6beab00ee0c756103 2023-04-18 22:53:50,686 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/a7b990a2771f4953ae61cd7f43f5ff31 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/a7b990a2771f4953ae61cd7f43f5ff31 2023-04-18 22:53:50,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681858430643, max timestamp=1681858430684Num uniques 1009; 2023-04-18 22:53:50,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681858430643, max timestamp=1681858430684Num uniques 1009; 2023-04-18 22:53:50,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:50,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:50,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858490687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858490684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=15386, suffix heap size=132904,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=336 2023-04-18 22:53:50,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=15386, suffix heap size=132904,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=336 2023-04-18 22:53:50,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681858430643, max timestamp=1681858430684Num uniques 1036; 2023-04-18 22:53:50,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 22:53:50,718 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/a7b990a2771f4953ae61cd7f43f5ff31, entries=150, sequenceid=10, filesize=11.6 K 2023-04-18 22:53:50,729 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/768745411809421f9ccadbe76b1b29d6 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/768745411809421f9ccadbe76b1b29d6 2023-04-18 22:53:50,775 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/768745411809421f9ccadbe76b1b29d6, entries=150, sequenceid=10, filesize=11.6 K 2023-04-18 22:53:50,779 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/fcc98a05222a4ce6beab00ee0c756103 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fcc98a05222a4ce6beab00ee0c756103 2023-04-18 22:53:50,801 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fcc98a05222a4ce6beab00ee0c756103, entries=150, sequenceid=10, filesize=11.6 K 2023-04-18 22:53:50,806 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=120.76 KB/123660 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 597ms, sequenceid=10, compaction requested=false 2023-04-18 22:53:50,808 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-18 22:53:50,810 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:50,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:50,812 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=134.18 KB heapSize=373.08 KB 2023-04-18 22:53:50,812 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:53:50,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:53:50,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:53:50,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:53:50,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:53:50,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:53:50,810 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:50,823 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:50,823 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:50,823 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,823 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:50,823 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:50,834 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:50,834 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:50,835 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:50,835 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:50,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:50,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2332f169[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:50,840 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:50,841 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:50,843 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:50,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:50,843 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:50,843 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:50,843 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:50,856 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:50,856 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:50,857 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:50,857 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,857 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:50,859 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:50,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:50,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858490851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:50,878 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:50,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:50,879 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:50,879 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:50,879 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:50,879 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:50,882 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:50,883 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:50,883 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:50,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:50,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:50,885 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/7f3eece9566b4492a86d7019498c3e7d 2023-04-18 22:53:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:50,886 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:50,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:50,890 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858490864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:50,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:50,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858490864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:50,908 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:50,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:50,909 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:50,909 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,909 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:50,909 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:50,909 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,909 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:50,909 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:50,909 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:50,909 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:50,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:50,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:50,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:50,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:50,912 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:50,912 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:50,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:50,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858490871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:50,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858490872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:50,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:50,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858490976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,012 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:51,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:51,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:51,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:51,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858490996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:51,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858490997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:51,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:51,036 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:51,036 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:51,037 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:51,037 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:51,038 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,038 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,040 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:51,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:51,043 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,044 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,044 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:51,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:51,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:51,045 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:51,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:51,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858491041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:51,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858491042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:51,056 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:51,056 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:51,056 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:51,056 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:51,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:51,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:51,059 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,059 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:51,059 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:51,066 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,069 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,069 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:51,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:51,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@246023e0[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:51,070 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:51,071 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,074 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,074 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:51,074 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:51,074 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:51,075 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,075 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:51,079 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:51,079 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:51,079 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:51,080 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:51,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:51,086 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:51,087 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:51,087 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:51,087 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:51,087 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:51,094 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,094 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,094 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:51,095 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:51,099 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,109 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,109 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,112 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:51,112 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:51,112 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:51,116 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,117 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:51,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:51,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:51,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:51,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:51,118 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:51,118 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:51,117 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,118 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,119 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:51,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858491201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,218 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:51,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:51,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:51,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:51,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:51,236 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:51,236 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:51,240 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:51,241 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:51,241 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,242 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,243 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,243 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:51,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:51,244 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,244 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:51,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:51,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:51,247 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:51,248 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:51,257 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:51,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:51,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858491243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,257 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:51,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:51,259 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:51,259 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:51,259 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:51,259 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:51,260 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:51,260 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:51,260 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,260 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:51,260 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:51,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:51,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858491243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,265 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,265 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:51,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:51,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:51,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e6d2d98[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:51,268 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:51,268 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:51,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858491261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:51,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858491262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,275 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,275 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:51,275 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:51,275 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:51,276 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,276 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:51,278 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:51,278 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:51,278 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:51,278 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:51,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:51,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:51,284 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:51,284 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:51,284 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:51,284 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:51,284 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:51,287 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,287 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,287 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:51,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:51,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:51,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,289 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,292 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:51,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,305 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,305 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,305 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,305 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,305 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:51,305 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:51,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:51,305 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:51,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:51,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:51,305 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,306 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,306 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:51,306 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:51,308 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:51,399 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/6e7ba278388d41f28e1eb92b608bb0d7 2023-04-18 22:53:51,408 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:51,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:51,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:51,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:51,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:51,470 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:51,471 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:51,490 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:51,490 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:51,491 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,491 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,494 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:51,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:51,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:51,495 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,495 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:51,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:51,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:51,500 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:51,501 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:51,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:51,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:51,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:51,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:51,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858491517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,521 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:51,521 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:51,521 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:51,521 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:51,527 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:51,527 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:51,527 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:51,531 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,532 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,532 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:51,540 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:51,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:51,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59f51a5[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:51,547 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:51,547 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,548 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,549 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:51,549 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:51,549 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,549 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:51,549 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:51,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:51,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858491560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,573 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:51,573 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:51,574 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:51,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:51,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858491577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:51,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858491579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,584 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:51,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:51,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858491580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:51,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:51,597 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:51,598 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:51,598 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:51,598 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:51,598 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:51,602 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,602 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:51,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:51,602 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:51,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:51,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:51,603 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,605 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:51,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,624 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:51,624 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:51,624 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,624 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,624 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,624 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,624 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:51,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:51,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:51,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:51,626 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:51,626 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:51,624 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,636 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,636 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:51,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:51,726 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:51,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:51,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:51,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:51,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:51,752 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:51,752 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:51,754 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:51,754 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:51,764 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,764 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,766 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:51,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:51,766 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:51,767 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,767 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:51,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:51,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:51,772 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:51,772 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:51,772 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:51,772 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:51,775 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:51,775 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:51,780 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,780 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:51,781 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,781 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:51,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:51,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@319251c6[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:51,784 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:51,784 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,786 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,786 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:51,786 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:51,786 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:51,786 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,786 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:51,788 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:51,788 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:51,788 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:51,790 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:51,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:51,801 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:51,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:51,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:51,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:51,801 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:51,806 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,806 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:51,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,806 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:51,806 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,809 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,819 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:51,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:51,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:51,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:51,819 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,820 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,820 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,820 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,820 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,820 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:51,820 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,821 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:51,821 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:51,820 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:51,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:51,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:51,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:51,905 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/08bb7a7e62c94ad78d1de72644081ebd 2023-04-18 22:53:51,924 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:51,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:51,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:51,938 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/7f3eece9566b4492a86d7019498c3e7d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/7f3eece9566b4492a86d7019498c3e7d 2023-04-18 22:53:51,953 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/7f3eece9566b4492a86d7019498c3e7d, entries=300, sequenceid=96, filesize=18.5 K 2023-04-18 22:53:51,956 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/6e7ba278388d41f28e1eb92b608bb0d7 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/6e7ba278388d41f28e1eb92b608bb0d7 2023-04-18 22:53:51,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:51,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:51,978 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:51,978 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:51,980 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:51,981 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:51,981 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,981 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,983 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:51,983 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:51,990 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:51,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,991 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:51,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:51,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:51,993 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/6e7ba278388d41f28e1eb92b608bb0d7, entries=150, sequenceid=96, filesize=11.6 K 2023-04-18 22:53:51,995 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/08bb7a7e62c94ad78d1de72644081ebd as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/08bb7a7e62c94ad78d1de72644081ebd 2023-04-18 22:53:52,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:52,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:52,005 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:52,005 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:52,006 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:52,008 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:52,008 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:52,008 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:52,008 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:52,011 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:52,012 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:52,013 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:52,013 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:52,013 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,013 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:52,013 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:52,021 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,022 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,022 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:52,024 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:52,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:52,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858492026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:52,030 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/08bb7a7e62c94ad78d1de72644081ebd, entries=150, sequenceid=96, filesize=11.6 K 2023-04-18 22:53:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,033 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~373.03 KB/381984, currentSize=46.96 KB/48090 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 1222ms, sequenceid=96, compaction requested=false 2023-04-18 22:53:52,034 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:52,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:52,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:52,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14481b37[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:52,035 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:52,035 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,038 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,038 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:52,038 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:52,038 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:52,038 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,038 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:52,040 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,040 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:52,040 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:52,040 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:52,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:52,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:52,060 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:52,060 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:52,060 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:52,060 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:52,060 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:52,063 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,063 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:52,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,063 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:52,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:52,065 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:52,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:52,068 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-18 22:53:52,068 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:53:52,068 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:52,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:53:52,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:52,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:53:52,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,076 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,082 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:52,082 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:52,082 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:52,082 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:52,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:52,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:52,082 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,089 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,089 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,089 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,089 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,089 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:52,089 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:52,089 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:52,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:52,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:52,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858492173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858432156, max timestamp=1681858432176Num uniques 859; 2023-04-18 22:53:52,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858432156, max timestamp=1681858432176Num uniques 859; 2023-04-18 22:53:52,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:53:52,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:52,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858432156, max timestamp=1681858432176Num uniques 859; 2023-04-18 22:53:52,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:53:52,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:53:52,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858492174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,189 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:52,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:52,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:52,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:52,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:52,200 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:52,200 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:52,205 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:52,205 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:52,206 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,206 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,208 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:52,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:52,208 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:52,209 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,209 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:52,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:52,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:52,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:52,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:52,212 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:52,212 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:52,213 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:52,214 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:52,214 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:52,214 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:52,215 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:52,215 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:52,215 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:52,217 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:52,217 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:52,217 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,217 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:52,217 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:52,219 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,219 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:52,220 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,220 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:52,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:52,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:52,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@732ef5cb[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:52,222 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:52,222 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,225 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,225 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:52,225 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:52,225 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:52,225 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,225 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:52,231 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,232 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:52,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:52,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:52,239 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:52,239 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:52,239 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:52,239 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:52,239 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:52,242 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,243 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:52,243 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:52,245 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:52,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,247 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:52,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,250 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:52,250 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:52,250 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:52,251 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:52,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:52,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:52,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:52,250 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,252 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:52,252 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:52,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:52,252 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,253 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,253 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,253 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,253 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:52,267 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-18 22:53:52,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:52,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858492282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:52,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858492283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:52,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858492283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:52,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858492285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,352 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:52,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:52,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:52,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:52,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:52,364 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:52,364 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:52,365 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:52,365 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:52,367 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,367 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,374 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,374 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:52,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:52,375 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,375 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:52,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:52,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:52,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:52,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:52,384 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:52,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:52,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858492387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:52,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858492392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,404 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:52,406 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:52,406 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:52,406 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:52,406 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:52,407 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:52,407 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:52,407 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:52,407 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,407 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:52,407 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:52,409 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,409 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:52,410 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,410 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:52,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:52,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:52,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:52,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:52,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cc39072[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:52,412 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:52,412 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,415 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,415 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-18 22:53:52,415 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:52,415 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:52,416 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,416 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:52,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:52,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:52,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:52,421 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:52,424 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-18 22:53:52,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:52,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:52,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:52,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:52,433 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:52,433 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:52,433 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:52,433 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:52,433 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:52,435 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,435 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:52,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,436 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:52,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,437 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:52,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:52,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,439 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,446 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:52,447 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,447 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:52,447 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:52,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:52,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:52,447 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,451 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,451 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,451 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,451 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,451 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:52,452 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:52,452 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:52,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:52,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:52,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:52,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:52,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858492491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858492491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,513 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/0984ce701d884f9cafc20b8d05976fe3 2023-04-18 22:53:52,552 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:52,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:52,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:52,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:52,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:52,581 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:52,581 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:52,583 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:52,583 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:52,589 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,589 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,594 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:52,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:52,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:52,594 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,595 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,595 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:52,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:52,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:52,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858492598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:52,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858492599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:52,617 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/b5d1369e93854d0889675ff2748275b5 2023-04-18 22:53:52,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:52,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:52,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:52,622 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:52,624 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:52,624 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:52,625 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:52,625 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:52,625 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:52,625 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:52,625 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:52,625 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:52,625 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,625 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:52,625 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:52,627 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,627 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ffcfc4d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:52,641 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:52,641 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,645 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,646 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:52,646 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:52,646 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,646 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:52,646 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:52,648 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:52,648 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,649 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:52,649 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:52,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:52,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:52,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:52,666 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:52,666 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:52,666 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:52,666 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:52,666 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:52,668 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,668 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,668 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:52,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:52,669 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,671 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:52,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,676 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,677 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,677 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,677 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,677 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,677 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,677 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,677 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:52,677 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:52,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:52,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:52,677 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:52,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:52,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:52,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:52,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:52,691 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:52,691 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:52,791 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:52,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:52,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:52,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858492796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:52,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858492798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:52,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:52,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:52,806 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:52,806 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:52,808 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:52,808 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:52,808 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,809 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,813 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:52,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:52,813 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,814 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,814 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:52,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:52,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:52,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:52,815 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:52,815 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:52,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:52,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:52,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:52,818 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:52,818 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:52,818 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:52,818 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:52,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:52,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:52,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:52,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:52,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:52,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:52,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:52,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b39db26[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:52,824 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:52,824 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,826 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,826 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:52,826 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:52,826 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:52,826 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,826 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:52,827 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:52,828 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:52,828 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:52,828 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:52,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:52,833 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:52,833 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:52,833 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:52,833 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:52,833 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:52,837 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,837 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,837 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:52,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:52,838 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:52,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,841 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,845 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,845 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:52,845 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:52,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:52,845 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:52,845 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,845 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,845 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,845 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:52,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:52,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:52,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:52,846 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:52,846 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:52,845 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:52,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:52,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858492905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:52,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858492905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,946 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:52,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:52,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:52,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:52,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:52,954 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:52,954 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:52,955 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:52,955 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:52,956 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,956 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,957 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,958 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:52,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:52,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:52,959 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,959 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:52,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:52,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:52,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:52,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:52,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:52,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:52,969 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:52,969 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:52,970 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:52,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:52,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:52,976 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:52,976 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:52,976 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:52,976 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:52,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:52,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:52,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,977 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:52,977 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:52,979 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,979 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:52,980 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,980 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:52,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:52,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fbde927[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:52,983 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:52,983 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,986 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,986 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,986 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:52,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:52,989 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:52,989 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:52,989 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:52,989 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:52,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:53,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:53,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:53,001 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:53,001 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:53,001 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:53,001 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:53,001 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:53,003 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,003 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:53,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,003 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:53,004 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:53,007 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:53,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,014 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:53,014 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:53,014 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:53,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:53,015 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:53,015 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:53,014 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:53,014 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:53,015 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,017 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,017 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,017 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,017 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:53,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:53,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:53,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:53,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858493038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,109 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/3e390e11a80f4551bf92115d78e87864 2023-04-18 22:53:53,115 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:53,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:53,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:53,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:53,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:53,124 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:53,124 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:53,125 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:53,125 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:53,127 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,127 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,130 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:53,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:53,131 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,132 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,132 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:53,133 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/0984ce701d884f9cafc20b8d05976fe3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0984ce701d884f9cafc20b8d05976fe3 2023-04-18 22:53:53,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:53,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:53,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:53,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:53,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:53,135 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:53,135 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:53,135 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:53,135 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:53,135 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:53,135 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:53,136 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:53,145 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,145 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:53,146 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,146 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:53,146 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0984ce701d884f9cafc20b8d05976fe3, entries=150, sequenceid=107, filesize=11.6 K 2023-04-18 22:53:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,151 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/b5d1369e93854d0889675ff2748275b5 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b5d1369e93854d0889675ff2748275b5 2023-04-18 22:53:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:53,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:53,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6db6a2[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:53,153 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:53,153 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,154 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:53,155 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,155 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:53,156 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:53,157 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:53,157 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:53,169 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,171 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b5d1369e93854d0889675ff2748275b5, entries=150, sequenceid=107, filesize=11.6 K 2023-04-18 22:53:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:53,185 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:53,185 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:53,185 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:53,185 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:53,186 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:53,188 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:53,188 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,188 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:53,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:53,191 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,192 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/3e390e11a80f4551bf92115d78e87864 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/3e390e11a80f4551bf92115d78e87864 2023-04-18 22:53:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:53,194 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,201 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:53,201 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:53,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:53,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:53,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:53,202 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,202 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,202 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,202 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:53,202 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,212 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,212 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,212 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:53,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:53,217 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:53,227 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:53,229 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/3e390e11a80f4551bf92115d78e87864, entries=150, sequenceid=107, filesize=11.6 K 2023-04-18 22:53:53,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:53,231 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 1162ms, sequenceid=107, compaction requested=true 2023-04-18 22:53:53,231 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:53,260 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:53:53,260 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:53:53,260 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:53:53,261 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:53:53,262 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:53:53,276 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:53:53,278 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/B is initiating minor compaction (all files) 2023-04-18 22:53:53,279 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/B in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:53,279 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/768745411809421f9ccadbe76b1b29d6, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/6e7ba278388d41f28e1eb92b608bb0d7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b5d1369e93854d0889675ff2748275b5] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=34.8 K 2023-04-18 22:53:53,281 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42678 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:53:53,281 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/A is initiating minor compaction (all files) 2023-04-18 22:53:53,281 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 768745411809421f9ccadbe76b1b29d6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681858430139 2023-04-18 22:53:53,285 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/A in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:53,288 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/a7b990a2771f4953ae61cd7f43f5ff31, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/7f3eece9566b4492a86d7019498c3e7d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0984ce701d884f9cafc20b8d05976fe3] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=41.7 K 2023-04-18 22:53:53,296 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 6e7ba278388d41f28e1eb92b608bb0d7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681858430789 2023-04-18 22:53:53,297 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting a7b990a2771f4953ae61cd7f43f5ff31, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681858430139 2023-04-18 22:53:53,307 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting b5d1369e93854d0889675ff2748275b5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1681858430850 2023-04-18 22:53:53,308 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 7f3eece9566b4492a86d7019498c3e7d, keycount=300, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681858430722 2023-04-18 22:53:53,317 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 0984ce701d884f9cafc20b8d05976fe3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1681858430850 2023-04-18 22:53:53,325 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-18 22:53:53,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:53,327 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:53,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:53,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681858432266, max timestamp=1681858433319Num uniques 1018; 2023-04-18 22:53:53,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-18 22:53:53,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681858432266, max timestamp=1681858433319Num uniques 1018; 2023-04-18 22:53:53,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681858432266, max timestamp=1681858433319Num uniques 1018; 2023-04-18 22:53:53,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-18 22:53:53,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-18 22:53:53,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:53,334 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:53:53,334 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:53:53,334 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:53:53,334 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:53:53,334 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:53:53,334 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:53:53,354 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#A#compaction#9 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:53:53,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:53,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:53,359 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:53,360 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:53,361 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:53,361 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:53,362 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,362 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,363 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:53,363 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:53,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:53,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:53,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:53,374 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:53,374 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:53,375 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,375 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:53,377 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:53,386 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#B#compaction#11 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:53:53,408 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:53,416 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:53,415 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:53,431 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:53,431 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:53,432 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:53,435 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:53,435 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:53,436 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,436 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:53,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:53,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:53,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@273b3196[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:53,438 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:53,438 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,440 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,440 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:53,440 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:53,440 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:53,440 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,440 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:53,501 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:53,502 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:53,502 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:53,513 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/aa78dcdf9ecf439a97d1d61ed121c6c9 2023-04-18 22:53:53,517 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:53,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:53,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:53,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:53,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:53,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:53,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:53,525 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:53,525 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:53,525 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:53,525 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:53,525 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:53,526 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/b9451068b4e34c4fa315bb24d0d1dc5b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/b9451068b4e34c4fa315bb24d0d1dc5b 2023-04-18 22:53:53,528 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:53,529 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,529 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:53,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:53,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:53,530 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,532 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,536 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,536 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,536 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,536 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:53,536 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:53,536 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:53,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:53,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:53,537 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:53,537 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:53,536 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:53,536 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,537 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,537 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:53,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:53,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858433451, max timestamp=1681858433523Num uniques 859; 2023-04-18 22:53:53,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:53:53,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858433451, max timestamp=1681858433502Num uniques 859; 2023-04-18 22:53:53,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858433451, max timestamp=1681858433523Num uniques 859; 2023-04-18 22:53:53,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:53:53,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:53:53,571 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/A of 3d5ba4e7a99f0fcc1148990ad0a52e36 into b9451068b4e34c4fa315bb24d0d1dc5b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:53:53,574 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:53,574 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/A, priority=13, startTime=1681858433233; duration=0sec 2023-04-18 22:53:53,575 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:53:53,575 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:53:53,578 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:53:53,578 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/C is initiating minor compaction (all files) 2023-04-18 22:53:53,578 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/C in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:53,578 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fcc98a05222a4ce6beab00ee0c756103, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/08bb7a7e62c94ad78d1de72644081ebd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/3e390e11a80f4551bf92115d78e87864] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=34.8 K 2023-04-18 22:53:53,580 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting fcc98a05222a4ce6beab00ee0c756103, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681858430139 2023-04-18 22:53:53,581 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 08bb7a7e62c94ad78d1de72644081ebd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681858430789 2023-04-18 22:53:53,583 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 3e390e11a80f4551bf92115d78e87864, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1681858430850 2023-04-18 22:53:53,621 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#C#compaction#13 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:53:53,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681858433583, max timestamp=1681858433616Num uniques 1018; 2023-04-18 22:53:53,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681858433583, max timestamp=1681858433616Num uniques 1018; 2023-04-18 22:53:53,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-18 22:53:53,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-18 22:53:53,629 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/e4b7ba5b0de4489bb6fd02f701051f77 2023-04-18 22:53:53,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681858433583, max timestamp=1681858433616Num uniques 1018; 2023-04-18 22:53:53,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-18 22:53:53,637 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:53,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:53,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:53,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:53,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:53,652 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:53,652 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:53,659 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:53,659 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:53,661 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,661 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:53,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:53,664 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,664 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,664 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:53,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:53,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:53,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:53,680 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:53,680 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:53,685 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:53,692 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:53,692 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:53,692 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:53,692 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:53,693 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:53,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:53,696 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:53,700 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,700 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:53,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:53,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12ee3ebc[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:53,704 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:53,704 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,706 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,706 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:53,706 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:53,706 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:53,710 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/c87dc38019124e97914e753aa5885ae9 2023-04-18 22:53:53,712 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:53,713 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,713 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:53,713 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:53,714 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:53,714 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:53,714 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:53,714 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:53,714 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:53,714 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:53,714 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:53,717 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,717 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:53,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:53,717 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:53,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,717 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:53,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:53,719 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:53,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858493712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:53,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681858433685, max timestamp=1681858433715Num uniques 1177; 2023-04-18 22:53:53,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=12682, suffix heap size=150376,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=277 2023-04-18 22:53:53,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681858433685, max timestamp=1681858433715Num uniques 1177; 2023-04-18 22:53:53,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=12682, suffix heap size=150376,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=277 2023-04-18 22:53:53,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681858433685, max timestamp=1681858433715Num uniques 1177; 2023-04-18 22:53:53,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=12682, suffix heap size=150376,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=277 2023-04-18 22:53:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,725 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,725 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,725 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,725 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:53,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:53,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:53,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:53,725 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:53,725 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,732 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,732 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,732 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,732 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:53,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:53,730 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:53,732 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:53,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:53,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:53,736 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/aa78dcdf9ecf439a97d1d61ed121c6c9 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/aa78dcdf9ecf439a97d1d61ed121c6c9 2023-04-18 22:53:53,747 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/aa78dcdf9ecf439a97d1d61ed121c6c9, entries=250, sequenceid=145, filesize=16.4 K 2023-04-18 22:53:53,749 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/e4b7ba5b0de4489bb6fd02f701051f77 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/e4b7ba5b0de4489bb6fd02f701051f77 2023-04-18 22:53:53,763 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/cbde6625cee14421924167fa988b5a62 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/cbde6625cee14421924167fa988b5a62 2023-04-18 22:53:53,785 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/e4b7ba5b0de4489bb6fd02f701051f77, entries=150, sequenceid=145, filesize=11.8 K 2023-04-18 22:53:53,796 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/c87dc38019124e97914e753aa5885ae9 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c87dc38019124e97914e753aa5885ae9 2023-04-18 22:53:53,798 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/C of 3d5ba4e7a99f0fcc1148990ad0a52e36 into cbde6625cee14421924167fa988b5a62(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:53:53,798 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:53,798 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/C, priority=13, startTime=1681858433260; duration=0sec 2023-04-18 22:53:53,801 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:53:53,814 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c87dc38019124e97914e753aa5885ae9, entries=150, sequenceid=145, filesize=11.8 K 2023-04-18 22:53:53,816 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~80.58 KB/82512, currentSize=100.63 KB/103050 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 491ms, sequenceid=145, compaction requested=false 2023-04-18 22:53:53,818 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:53,821 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-18 22:53:53,821 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-18 22:53:53,826 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=107.34 KB heapSize=294.23 KB 2023-04-18 22:53:53,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:53,828 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:53:53,829 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:53:53,829 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:53:53,829 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:53:53,829 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:53:53,829 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:53:53,832 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:53,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:53,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:53,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:53,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:53,850 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:53,850 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:53,854 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:53,854 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:53,855 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,855 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,857 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,857 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:53,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:53,858 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,858 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:53,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:53,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:53,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:53,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:53,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:53,863 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:53,868 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:53,868 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:53,868 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:53,868 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:53,868 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:53,868 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:53,868 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:53,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858493866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:53,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858493866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:53,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858493867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:53,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858493869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,884 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:53,884 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:53,893 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:53,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:53,893 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:53,894 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,894 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:53,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:53,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:53,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:53,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20a1917[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:53,896 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:53,896 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,901 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:53,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:53,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:53,904 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,904 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:53,905 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:53,905 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:53,905 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:53,906 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:53,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:53,912 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:53,912 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:53,912 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:53,912 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:53,912 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:53,914 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,914 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:53,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,914 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:53,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,916 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:53,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,919 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,924 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:53,924 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:53,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:53,924 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:53,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:53,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:53,925 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:53,925 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:53,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:53,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:53,925 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,925 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,926 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:53,926 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,926 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,926 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:53,926 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:53,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:53,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858493979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:53,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858493979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:53,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858493979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:53,990 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/924701d5366d4f03a8005978357e0c57 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/924701d5366d4f03a8005978357e0c57 2023-04-18 22:53:53,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:53,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858493988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,003 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/B of 3d5ba4e7a99f0fcc1148990ad0a52e36 into 924701d5366d4f03a8005978357e0c57(size=11.7 K), total size for store is 23.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:53:54,003 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:54,004 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/B, priority=13, startTime=1681858433260; duration=0sec 2023-04-18 22:53:54,004 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:53:54,025 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:54,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:54,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:54,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:54,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:54,031 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:54,031 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:54,032 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:54,032 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:54,033 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,033 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,035 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:54,035 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:54,035 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,035 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:54,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:54,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:54,037 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:54,037 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:54,038 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:54,039 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:54,039 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:54,039 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:54,039 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:54,039 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:54,039 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,039 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:54,039 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:54,041 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,041 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:54,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:54,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:54,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:54,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d06752[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:54,046 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:54,046 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,047 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:54,048 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,048 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:54,050 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:54,051 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:54,051 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:54,051 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:54,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:54,057 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:54,057 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:54,057 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:54,057 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:54,057 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:54,067 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,067 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:54,068 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:54,068 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:54,071 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,075 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:54,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:54,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:54,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:54,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:54,076 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,076 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,076 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:54,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:54,077 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:54,077 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:54,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:54,177 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:54,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:54,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:54,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:54,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858494183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:54,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858494183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:54,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:54,195 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:54,195 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:54,201 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:54,201 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:54,201 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,201 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,203 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:54,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:54,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:54,203 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:54,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,204 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,205 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:54,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858494189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:54,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:54,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:54,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:54,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858494211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,216 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:54,216 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:54,219 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:54,220 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:54,220 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:54,220 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:54,220 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:54,220 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:54,220 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:54,220 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:54,220 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:54,220 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,221 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:54,221 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:54,224 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,224 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:54,225 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,225 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:54,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:54,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:54,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e99515f[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:54,233 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:54,233 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,246 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,246 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:54,246 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:54,246 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:54,251 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,251 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:54,251 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,251 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:54,251 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:54,251 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:54,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:54,269 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:54,269 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:54,269 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:54,269 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:54,269 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:54,271 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:54,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,272 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:54,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,277 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:54,284 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:54,285 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:54,285 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,285 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,285 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:54,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:54,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:54,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:54,285 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,286 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,286 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,286 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,287 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,287 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:54,287 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:54,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:54,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:54,307 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/435d66c40bb5496eb34dfdd6801a8b4a 2023-04-18 22:53:54,357 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/8265452985964572905a8c0112e9a36c 2023-04-18 22:53:54,387 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:54,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:54,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:54,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:54,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:54,420 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:54,420 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:54,427 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:54,427 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:54,428 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,428 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,429 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,429 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:54,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:54,430 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,430 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:54,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:54,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:54,435 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:54,435 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:54,436 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:54,468 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:54,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:54,471 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:54,471 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:54,471 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:54,471 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:54,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:54,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:54,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:54,474 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:54,478 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,478 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:54,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:54,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1de18bee[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:54,481 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:54,481 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,483 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,483 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:54,483 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:54,483 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:54,484 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,484 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:54,486 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,486 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:54,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:54,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858494489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:54,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:54,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858494491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:54,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:54,498 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:54,498 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:54,498 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:54,498 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:54,498 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:54,502 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,503 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:54,503 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:54,503 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:54,507 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:54,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,510 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:54,510 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:54,510 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:54,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:54,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:54,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:54,510 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,520 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,520 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:54,520 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:54,520 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:54,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858494510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:54,520 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:54,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:54,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:54,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858494516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,621 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:54,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:54,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:54,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:54,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:54,640 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:54,640 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:54,645 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:54,645 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:54,647 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,647 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,649 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:54,649 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:54,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,650 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,650 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:54,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:54,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:54,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:54,660 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:54,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:54,661 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:54,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:54,661 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:54,661 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:54,661 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:54,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:54,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:54,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:54,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:54,664 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,664 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:54,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:54,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f71fbe4[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:54,669 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:54,669 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,670 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,670 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:54,670 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:54,670 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:54,671 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,671 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:54,672 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:54,672 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:54,672 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,672 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:54,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:54,684 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:54,684 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:54,684 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:54,684 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:54,684 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:54,686 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,686 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:54,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,686 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:54,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:54,688 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,690 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,693 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,693 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,693 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,693 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,693 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:54,693 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:54,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:54,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:54,693 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:54,693 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:54,694 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,694 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:54,698 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:54,699 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:54,799 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:54,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:54,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:54,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:54,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:54,813 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:54,813 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:54,814 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:54,814 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:54,815 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,815 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,816 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,816 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:54,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:54,818 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,818 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:54,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:54,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:54,827 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:54,827 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:54,833 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:54,834 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:54,834 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:54,834 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:54,834 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:54,834 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:54,834 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:54,838 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:54,838 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:54,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:54,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c18ffaf[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:54,842 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:54,842 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,842 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,842 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:54,843 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,844 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,844 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:54,844 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:54,844 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:54,844 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:54,846 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:54,846 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:54,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,846 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:54,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:54,852 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:54,852 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:54,852 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:54,852 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:54,852 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:54,856 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,856 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:54,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,857 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:54,859 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,862 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,867 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:54,867 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:54,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:54,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:54,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:54,868 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:54,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:54,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:54,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:54,869 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,869 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:54,869 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:54,869 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:54,869 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:54,876 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/f996372401674774bfee37c55ff3732a 2023-04-18 22:53:54,890 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/435d66c40bb5496eb34dfdd6801a8b4a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/435d66c40bb5496eb34dfdd6801a8b4a 2023-04-18 22:53:54,899 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/435d66c40bb5496eb34dfdd6801a8b4a, entries=200, sequenceid=213, filesize=14.1 K 2023-04-18 22:53:54,912 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/8265452985964572905a8c0112e9a36c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8265452985964572905a8c0112e9a36c 2023-04-18 22:53:54,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8265452985964572905a8c0112e9a36c, entries=150, sequenceid=213, filesize=11.8 K 2023-04-18 22:53:54,950 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/f996372401674774bfee37c55ff3732a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f996372401674774bfee37c55ff3732a 2023-04-18 22:53:54,963 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f996372401674774bfee37c55ff3732a, entries=150, sequenceid=213, filesize=11.8 K 2023-04-18 22:53:54,969 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:54,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:54,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:54,982 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~314.11 KB/321648, currentSize=67.09 KB/68700 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 1156ms, sequenceid=213, compaction requested=true 2023-04-18 22:53:54,982 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:54,983 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:53:54,983 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:53:54,983 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:53:54,983 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:53:54,983 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:53:54,985 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 43231 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:53:54,986 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/A is initiating minor compaction (all files) 2023-04-18 22:53:54,986 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36061 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:53:54,986 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/A in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:54,986 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/B is initiating minor compaction (all files) 2023-04-18 22:53:54,986 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/B in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:54,986 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/b9451068b4e34c4fa315bb24d0d1dc5b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/aa78dcdf9ecf439a97d1d61ed121c6c9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/435d66c40bb5496eb34dfdd6801a8b4a] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=42.2 K 2023-04-18 22:53:54,986 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/924701d5366d4f03a8005978357e0c57, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/e4b7ba5b0de4489bb6fd02f701051f77, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8265452985964572905a8c0112e9a36c] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=35.2 K 2023-04-18 22:53:54,987 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting b9451068b4e34c4fa315bb24d0d1dc5b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1681858430850 2023-04-18 22:53:54,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:55,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:54,987 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 924701d5366d4f03a8005978357e0c57, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1681858430850 2023-04-18 22:53:55,001 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:55,001 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:55,001 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:55,001 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:55,002 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,002 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,003 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting aa78dcdf9ecf439a97d1d61ed121c6c9, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681858432266 2023-04-18 22:53:55,004 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 435d66c40bb5496eb34dfdd6801a8b4a, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681858433803 2023-04-18 22:53:55,004 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:55,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,006 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,006 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,006 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:55,007 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting e4b7ba5b0de4489bb6fd02f701051f77, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681858432282 2023-04-18 22:53:55,008 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 22:53:55,008 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 8265452985964572905a8c0112e9a36c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681858433808 2023-04-18 22:53:55,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:55,008 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:53:55,008 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:55,008 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:53:55,009 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:55,009 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:53:55,009 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:55,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:55,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:55,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:55,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:55,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:55,031 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:55,031 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:55,031 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:55,031 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:55,031 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:55,031 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:55,031 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:55,031 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:55,031 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,056 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#B#compaction#18 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:53:55,058 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#A#compaction#19 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:53:55,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858495054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858495054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,068 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:55,068 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:55,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858495059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858495059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858495059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,077 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,105 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,105 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:55,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:55,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@783f4cda[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:55,108 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:55,108 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,109 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,109 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:55,109 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:55,109 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:55,109 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,110 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:55,112 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,113 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:55,113 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:55,113 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:55,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:55,131 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:55,131 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:55,131 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:55,131 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:55,131 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:55,134 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,134 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:55,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,134 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:55,135 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,151 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,158 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/58506a812ab042b695f267284e762257 2023-04-18 22:53:55,162 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/38945d35cd61473895b55b26dc59a9f5 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/38945d35cd61473895b55b26dc59a9f5 2023-04-18 22:53:55,163 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:55,163 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:55,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:55,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:55,164 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,164 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,164 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,164 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,164 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,164 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:55,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:55,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:55,164 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:55,164 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:55,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:55,164 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,165 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:55,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858495166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858495167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,179 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/A of 3d5ba4e7a99f0fcc1148990ad0a52e36 into 38945d35cd61473895b55b26dc59a9f5(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:53:55,179 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:55,179 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/A, priority=13, startTime=1681858434983; duration=0sec 2023-04-18 22:53:55,180 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:53:55,180 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:53:55,182 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36061 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:53:55,182 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/C is initiating minor compaction (all files) 2023-04-18 22:53:55,182 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/C in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:55,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,182 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/cbde6625cee14421924167fa988b5a62, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c87dc38019124e97914e753aa5885ae9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f996372401674774bfee37c55ff3732a] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=35.2 K 2023-04-18 22:53:55,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858495178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,183 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting cbde6625cee14421924167fa988b5a62, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1681858430850 2023-04-18 22:53:55,187 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting c87dc38019124e97914e753aa5885ae9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681858432282 2023-04-18 22:53:55,188 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting f996372401674774bfee37c55ff3732a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681858433808 2023-04-18 22:53:55,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858495180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858495197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,221 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/0aeef7df8f414762b0c4c53ca8a424ae 2023-04-18 22:53:55,222 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#C#compaction#22 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:53:55,264 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:55,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:55,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:55,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:55,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:55,272 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:55,272 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:55,272 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:55,272 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:55,272 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,273 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,274 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:55,274 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,275 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,275 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:55,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:55,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:55,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:55,285 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:55,285 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:55,285 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:55,286 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:55,304 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:55,304 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:55,304 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:55,304 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:55,305 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:55,305 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:55,305 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:55,305 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,308 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:55,308 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:55,310 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:55,315 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/72c212879d28493b98bbeba2452f7b63 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/72c212879d28493b98bbeba2452f7b63 2023-04-18 22:53:55,315 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,315 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:55,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:55,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@691006c8[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:55,319 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:55,319 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,321 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,321 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:55,323 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:55,323 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:55,325 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,325 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:55,327 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:55,327 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:55,328 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:55,328 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:55,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,331 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/C of 3d5ba4e7a99f0fcc1148990ad0a52e36 into 72c212879d28493b98bbeba2452f7b63(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:53:55,331 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,331 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/C, priority=13, startTime=1681858434983; duration=0sec 2023-04-18 22:53:55,331 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/197e5f72da9e45d3b5d3a848072fc86d 2023-04-18 22:53:55,331 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:53:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:55,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:55,337 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:55,338 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:55,338 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:55,338 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:55,338 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:55,340 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,340 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:55,340 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:55,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,342 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:55,344 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,346 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/58506a812ab042b695f267284e762257 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/58506a812ab042b695f267284e762257 2023-04-18 22:53:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,348 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,348 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:55,348 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:55,348 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,348 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:55,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:55,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:55,348 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:55,348 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:55,349 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:55,348 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:55,349 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,349 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,349 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:55,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:55,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/58506a812ab042b695f267284e762257, entries=200, sequenceid=228, filesize=14.1 K 2023-04-18 22:53:55,360 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/0aeef7df8f414762b0c4c53ca8a424ae as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/0aeef7df8f414762b0c4c53ca8a424ae 2023-04-18 22:53:55,375 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/0aeef7df8f414762b0c4c53ca8a424ae, entries=150, sequenceid=228, filesize=11.8 K 2023-04-18 22:53:55,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858495371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,383 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/197e5f72da9e45d3b5d3a848072fc86d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/197e5f72da9e45d3b5d3a848072fc86d 2023-04-18 22:53:55,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858495383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858495385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,402 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/197e5f72da9e45d3b5d3a848072fc86d, entries=150, sequenceid=228, filesize=11.8 K 2023-04-18 22:53:55,404 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 397ms, sequenceid=228, compaction requested=false 2023-04-18 22:53:55,404 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:55,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:55,411 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 22:53:55,411 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:53:55,411 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:55,411 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:53:55,411 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:55,411 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:53:55,411 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:55,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,449 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:55,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858495446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:55,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:55,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858495449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:55,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:55,464 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:55,464 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:55,464 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:55,464 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:55,465 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,465 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,466 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:55,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:55,466 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,467 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,467 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:55,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:55,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:55,475 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:55,475 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:55,475 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:55,476 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:55,476 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:55,476 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:55,476 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:55,476 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:55,476 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:55,476 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:55,476 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:55,476 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,476 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:55,476 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:55,479 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,479 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:55,480 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,480 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:55,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:55,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29fe6ed5[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:55,481 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:55,481 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,482 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:55,483 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:55,483 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:55,484 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:55,484 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:55,484 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:55,484 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,490 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:55,490 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:55,490 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:55,490 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:55,490 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:55,490 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:55,490 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:55,494 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,494 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:55,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,496 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:55,496 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:55,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,498 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,506 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:55,506 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:55,506 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:55,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:55,506 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,506 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:55,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:55,506 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,507 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,507 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,507 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:55,507 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:55,507 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:55,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:55,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:55,541 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/df6b205a4c6346f7bba1ca9906fcce32 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/df6b205a4c6346f7bba1ca9906fcce32 2023-04-18 22:53:55,553 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/B of 3d5ba4e7a99f0fcc1148990ad0a52e36 into df6b205a4c6346f7bba1ca9906fcce32(size=12.0 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:53:55,553 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:55,553 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/B, priority=13, startTime=1681858434983; duration=0sec 2023-04-18 22:53:55,553 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:53:55,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858495552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858495555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,607 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:55,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:55,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:55,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:55,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:55,613 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:55,613 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:55,614 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:55,614 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:55,614 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,614 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,616 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,616 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:55,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:55,617 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,617 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:55,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:55,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:55,620 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:55,620 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:55,620 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:55,621 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:55,621 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:55,621 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:55,621 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:55,621 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,621 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:55,621 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:55,622 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:55,623 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:55,624 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,624 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:55,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:55,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@229019c2[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:55,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:55,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,626 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,627 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:55,627 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,627 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:55,627 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:55,627 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:55,631 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:55,631 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,631 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:55,631 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:55,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:55,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:55,634 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:55,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:55,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:55,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:55,634 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:55,635 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,635 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:55,635 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:55,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:55,636 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,638 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:55,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,645 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,645 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:55,645 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:55,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:55,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:55,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:55,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:55,645 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:55,646 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:55,646 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:55,645 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,645 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:55,646 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:55,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:55,646 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,646 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,646 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858495686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858495699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858495701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,746 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:55,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:55,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:55,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:55,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:55,753 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:55,753 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:55,753 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:55,753 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:55,753 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,753 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,756 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:55,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:55,756 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:55,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:55,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:55,758 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:55,758 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:55,758 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:55,758 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:55,758 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:55,759 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:55,759 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,759 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:55,759 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:55,759 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,759 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:55,760 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,760 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:55,761 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,761 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:55,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:55,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:55,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:55,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858495755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:55,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@204bd8d6[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:55,763 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:55,763 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,764 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:55,765 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:55,765 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,765 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:55,767 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,767 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:55,767 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:55,767 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:55,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:55,770 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:55,770 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:55,770 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:55,770 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:55,770 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:55,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:55,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858495766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,778 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:55,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:55,779 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,780 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,780 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:55,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:55,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:55,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,783 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,786 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:55,786 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:55,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:55,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:55,787 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,787 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,787 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,788 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,788 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,788 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,788 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,788 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:55,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:55,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:55,788 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:55,788 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:55,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:55,843 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/4ff19fef60a54d00b139036e8ba78aaf 2023-04-18 22:53:55,866 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/8679216824a0471786cca255f5c5f6dd 2023-04-18 22:53:55,888 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:55,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:55,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:55,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:55,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:55,904 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:55,904 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:55,904 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:55,904 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:55,905 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,905 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,906 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/828f87cadcfd4236a4efa39422ca50ca 2023-04-18 22:53:55,906 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:55,906 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,907 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,907 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:55,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:55,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:55,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:55,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:55,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:55,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:55,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:55,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:55,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:55,909 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:55,909 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:55,909 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:55,909 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:55,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:55,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:55,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:55,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:55,912 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:55,912 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:55,913 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,913 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:55,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:55,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4904ac51[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:55,915 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:55,915 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,916 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/4ff19fef60a54d00b139036e8ba78aaf as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/4ff19fef60a54d00b139036e8ba78aaf 2023-04-18 22:53:55,916 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,917 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:55,917 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:55,917 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:55,917 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,917 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:55,919 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:55,919 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:55,919 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:55,919 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,924 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/4ff19fef60a54d00b139036e8ba78aaf, entries=150, sequenceid=249, filesize=11.8 K 2023-04-18 22:53:55,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:55,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:55,924 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:55,925 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:55,925 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:55,926 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:55,926 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:55,927 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/8679216824a0471786cca255f5c5f6dd as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8679216824a0471786cca255f5c5f6dd 2023-04-18 22:53:55,928 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,928 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:55,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,928 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:55,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:55,929 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,934 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,937 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8679216824a0471786cca255f5c5f6dd, entries=150, sequenceid=249, filesize=11.8 K 2023-04-18 22:53:55,938 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:55,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:55,939 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,939 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,939 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:55,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:55,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:55,939 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:55,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:55,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:55,940 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:55,941 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:55,941 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,941 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:55,941 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:55,941 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:55,941 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/828f87cadcfd4236a4efa39422ca50ca as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/828f87cadcfd4236a4efa39422ca50ca 2023-04-18 22:53:55,941 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:55,950 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/828f87cadcfd4236a4efa39422ca50ca, entries=150, sequenceid=249, filesize=11.8 K 2023-04-18 22:53:55,954 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 542ms, sequenceid=249, compaction requested=true 2023-04-18 22:53:55,954 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:55,954 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:53:55,954 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:53:55,954 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:53:55,954 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:53:55,954 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:53:55,958 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38704 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:53:55,959 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/A is initiating minor compaction (all files) 2023-04-18 22:53:55,959 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:53:55,959 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/A in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:55,959 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/B is initiating minor compaction (all files) 2023-04-18 22:53:55,959 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/B in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:55,959 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/df6b205a4c6346f7bba1ca9906fcce32, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/0aeef7df8f414762b0c4c53ca8a424ae, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8679216824a0471786cca255f5c5f6dd] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=35.5 K 2023-04-18 22:53:55,959 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/38945d35cd61473895b55b26dc59a9f5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/58506a812ab042b695f267284e762257, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/4ff19fef60a54d00b139036e8ba78aaf] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=37.8 K 2023-04-18 22:53:55,960 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 38945d35cd61473895b55b26dc59a9f5, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681858433808 2023-04-18 22:53:55,960 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting df6b205a4c6346f7bba1ca9906fcce32, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681858433808 2023-04-18 22:53:55,962 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 58506a812ab042b695f267284e762257, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1681858433858 2023-04-18 22:53:55,962 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 0aeef7df8f414762b0c4c53ca8a424ae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1681858433858 2023-04-18 22:53:55,962 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 4ff19fef60a54d00b139036e8ba78aaf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681858435043 2023-04-18 22:53:55,962 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 8679216824a0471786cca255f5c5f6dd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681858435043 2023-04-18 22:53:55,986 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#A#compaction#27 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:53:55,993 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#B#compaction#28 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:53:56,041 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:56,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:56,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:56,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:56,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:56,048 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:56,048 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:56,048 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:56,048 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:56,048 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,049 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:56,050 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:56,050 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:56,051 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,051 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:56,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:56,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:56,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:56,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:56,054 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:56,055 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:56,055 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:56,055 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 22:53:56,055 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:53:56,056 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:56,060 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:53:56,060 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:56,060 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:53:56,060 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:56,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:56,067 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:56,117 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/c7940a10b5d84ee79ffcc91167afed94 2023-04-18 22:53:56,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:56,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858496146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:56,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858496148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:56,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858496189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:56,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858496209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:56,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858496212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:56,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858496253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:56,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858496254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,454 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/90f0fc2dc8e0488e9bff2abc2b8c5e6e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/90f0fc2dc8e0488e9bff2abc2b8c5e6e 2023-04-18 22:53:56,456 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/b582ac993d4a493280fda7f54a3c1431 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b582ac993d4a493280fda7f54a3c1431 2023-04-18 22:53:56,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:56,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858496459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,466 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/A of 3d5ba4e7a99f0fcc1148990ad0a52e36 into 90f0fc2dc8e0488e9bff2abc2b8c5e6e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:53:56,466 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/B of 3d5ba4e7a99f0fcc1148990ad0a52e36 into b582ac993d4a493280fda7f54a3c1431(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:53:56,466 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:56,467 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:56,467 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/A, priority=13, startTime=1681858435954; duration=0sec 2023-04-18 22:53:56,467 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/B, priority=13, startTime=1681858435954; duration=0sec 2023-04-18 22:53:56,467 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:53:56,467 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:53:56,467 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:53:56,469 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:53:56,469 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/C is initiating minor compaction (all files) 2023-04-18 22:53:56,469 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/C in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:56,469 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/72c212879d28493b98bbeba2452f7b63, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/197e5f72da9e45d3b5d3a848072fc86d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/828f87cadcfd4236a4efa39422ca50ca] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=35.5 K 2023-04-18 22:53:56,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:56,470 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 72c212879d28493b98bbeba2452f7b63, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681858433808 2023-04-18 22:53:56,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858496464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,472 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 197e5f72da9e45d3b5d3a848072fc86d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1681858433858 2023-04-18 22:53:56,474 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 828f87cadcfd4236a4efa39422ca50ca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681858435043 2023-04-18 22:53:56,525 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#C#compaction#31 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:53:56,543 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/086d26839d374229b8c0b82385944cd6 2023-04-18 22:53:56,563 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/6c76f07713c14c2abe68e12f67ff21c2 2023-04-18 22:53:56,570 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/c7940a10b5d84ee79ffcc91167afed94 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c7940a10b5d84ee79ffcc91167afed94 2023-04-18 22:53:56,579 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c7940a10b5d84ee79ffcc91167afed94, entries=150, sequenceid=263, filesize=11.9 K 2023-04-18 22:53:56,582 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/086d26839d374229b8c0b82385944cd6 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/086d26839d374229b8c0b82385944cd6 2023-04-18 22:53:56,591 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/086d26839d374229b8c0b82385944cd6, entries=150, sequenceid=263, filesize=11.9 K 2023-04-18 22:53:56,596 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/6c76f07713c14c2abe68e12f67ff21c2 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6c76f07713c14c2abe68e12f67ff21c2 2023-04-18 22:53:56,609 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6c76f07713c14c2abe68e12f67ff21c2, entries=150, sequenceid=263, filesize=11.9 K 2023-04-18 22:53:56,616 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 561ms, sequenceid=263, compaction requested=false 2023-04-18 22:53:56,616 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:56,617 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:56,617 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:56,618 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:56,618 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,618 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:56,618 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:56,621 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,621 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:56,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:56,623 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,623 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:56,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:56,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13d6fb62[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:56,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:56,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,627 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,628 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,632 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:56,635 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:56,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:56,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:56,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:56,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:56,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:56,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:56,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:56,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:56,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:56,642 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:56,642 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:56,642 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:56,642 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:56,642 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:56,645 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,645 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:56,645 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:56,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:56,646 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,649 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,654 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,654 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:56,654 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:56,654 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,654 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:56,655 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:56,654 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:56,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:56,654 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:56,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:56,655 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,655 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,655 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:56,655 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:56,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:56,755 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:56,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:56,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:56,774 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-18 22:53:56,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:53:56,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:53:56,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:53:56,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:53:56,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:53:56,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:53:56,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:56,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:56,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:56,780 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:56,780 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:56,781 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:56,781 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:56,781 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,781 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:56,782 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:56,782 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:56,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,784 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,784 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:56,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:56,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:56,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:56,796 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:56,796 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:56,796 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:56,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:56,804 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:56,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:56,804 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:56,804 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:56,804 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:56,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:56,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:56,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:56,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858496806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:56,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858496807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,824 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:56,824 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:56,831 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,831 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:56,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:56,832 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,832 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:56,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:56,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:56,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@312abed3[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:56,833 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:56,834 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,837 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,837 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:56,837 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:56,837 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:56,838 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,838 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:56,839 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,839 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:56,839 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:56,839 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:56,841 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/0e4cbfb051144576861cd409c9293b3d 2023-04-18 22:53:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:56,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:56,844 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:56,844 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:56,844 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:56,845 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:56,845 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:56,847 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:56,847 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,847 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:56,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:56,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:56,848 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,850 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,854 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,854 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:56,854 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:56,854 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,854 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,854 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,854 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,854 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:56,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:56,854 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:56,854 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:56,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:56,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:56,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:56,854 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:56,854 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:56,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:56,867 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/b752795d27ed48659d358df9450cf559 2023-04-18 22:53:56,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:56,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858496911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:56,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858496912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,950 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/faf1319830c1433a995430f9ca9e3261 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/faf1319830c1433a995430f9ca9e3261 2023-04-18 22:53:56,955 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:56,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:56,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:56,966 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/C of 3d5ba4e7a99f0fcc1148990ad0a52e36 into faf1319830c1433a995430f9ca9e3261(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:53:56,967 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:56,967 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/C, priority=13, startTime=1681858435954; duration=0sec 2023-04-18 22:53:56,967 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:53:56,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:56,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:56,971 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:56,971 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:56,971 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:56,971 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:56,972 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,972 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:56,973 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:56,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:56,974 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:56,974 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,974 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:56,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:56,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:56,975 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:56,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:56,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:56,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:56,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:56,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:56,980 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:56,980 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:56,980 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:56,980 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:56,980 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:56,980 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:56,980 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:56,980 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:56,980 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,980 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:56,980 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:56,982 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,982 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:56,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:56,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:56,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:56,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:56,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f595d48[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:56,985 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:56,985 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,987 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:56,988 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,988 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:56,989 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:56,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:56,992 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:56,992 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:56,992 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:56,992 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:56,992 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:56,995 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,995 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,995 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:56,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:56,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,006 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,008 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,011 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:57,011 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:57,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:57,011 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,012 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,012 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,012 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,012 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,012 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,012 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,012 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:57,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:57,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:57,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:57,012 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:57,013 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:57,113 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:57,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:57,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:57,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858497116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:57,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:57,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858497117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:57,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:57,132 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:57,132 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:57,132 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:57,132 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:57,133 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,133 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,134 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:57,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:57,134 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:57,135 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,135 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:57,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:57,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:57,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:57,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:57,144 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:57,144 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:57,144 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:57,144 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:57,144 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:57,144 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:57,144 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:57,144 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:57,144 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:57,144 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:57,145 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:57,145 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,145 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:57,145 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:57,146 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,146 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,147 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,147 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23d306d6[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:57,149 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:57,149 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,150 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:57,151 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:57,151 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,151 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:57,154 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:57,154 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,154 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:57,154 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:57,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:57,158 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:57,158 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:57,158 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:57,158 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:57,158 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:57,160 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,160 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,160 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:57,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:57,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:57,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,162 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,163 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,167 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:57,167 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:57,167 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,167 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,167 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,167 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:57,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:57,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:57,167 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,167 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,167 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,167 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:57,167 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:57,167 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:57,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:57,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:57,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:57,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858497197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:57,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858497221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:57,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858497223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,268 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:57,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:57,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:57,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:57,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:57,278 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:57,278 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:57,278 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:57,278 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:57,279 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,279 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,282 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:57,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:57,282 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,283 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,283 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:57,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:57,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:57,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:57,288 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:57,288 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:57,288 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:57,294 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:57,294 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:57,294 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/c03dba2a81604adfbce9276254a1e03f 2023-04-18 22:53:57,294 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:57,294 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:57,294 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:57,295 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:57,295 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:57,295 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:57,295 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,295 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:57,295 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:57,297 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,297 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,297 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,297 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:57,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:57,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fae425[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:57,303 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:57,303 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,304 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/0e4cbfb051144576861cd409c9293b3d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0e4cbfb051144576861cd409c9293b3d 2023-04-18 22:53:57,310 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,310 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:57,312 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:57,312 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0e4cbfb051144576861cd409c9293b3d, entries=150, sequenceid=286, filesize=11.9 K 2023-04-18 22:53:57,313 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/b752795d27ed48659d358df9450cf559 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b752795d27ed48659d358df9450cf559 2023-04-18 22:53:57,314 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:57,314 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:57,314 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,314 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:57,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:57,320 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:57,320 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:57,320 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:57,320 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:57,321 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:57,324 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,324 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,324 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:57,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:57,326 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,330 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b752795d27ed48659d358df9450cf559, entries=150, sequenceid=286, filesize=11.9 K 2023-04-18 22:53:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,332 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/c03dba2a81604adfbce9276254a1e03f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c03dba2a81604adfbce9276254a1e03f 2023-04-18 22:53:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,335 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,335 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,335 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,335 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:57,335 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,335 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,336 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,336 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:57,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:57,336 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:57,336 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:57,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:57,336 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:57,336 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:57,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:57,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:57,340 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c03dba2a81604adfbce9276254a1e03f, entries=150, sequenceid=286, filesize=11.9 K 2023-04-18 22:53:57,341 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=60.38 KB/61830 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 567ms, sequenceid=286, compaction requested=true 2023-04-18 22:53:57,341 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:57,341 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:53:57,341 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:53:57,341 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:53:57,341 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:53:57,345 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:53:57,347 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36666 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:53:57,347 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/A is initiating minor compaction (all files) 2023-04-18 22:53:57,347 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/A in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:57,347 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/90f0fc2dc8e0488e9bff2abc2b8c5e6e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c7940a10b5d84ee79ffcc91167afed94, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0e4cbfb051144576861cd409c9293b3d] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=35.8 K 2023-04-18 22:53:57,348 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36666 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:53:57,348 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 90f0fc2dc8e0488e9bff2abc2b8c5e6e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681858435043 2023-04-18 22:53:57,349 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/B is initiating minor compaction (all files) 2023-04-18 22:53:57,349 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/B in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:57,349 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b582ac993d4a493280fda7f54a3c1431, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/086d26839d374229b8c0b82385944cd6, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b752795d27ed48659d358df9450cf559] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=35.8 K 2023-04-18 22:53:57,350 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting c7940a10b5d84ee79ffcc91167afed94, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1681858435420 2023-04-18 22:53:57,350 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting b582ac993d4a493280fda7f54a3c1431, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681858435043 2023-04-18 22:53:57,351 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 0e4cbfb051144576861cd409c9293b3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681858436105 2023-04-18 22:53:57,351 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 086d26839d374229b8c0b82385944cd6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1681858435420 2023-04-18 22:53:57,352 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting b752795d27ed48659d358df9450cf559, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681858436105 2023-04-18 22:53:57,369 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#A#compaction#36 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:53:57,374 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#B#compaction#37 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:53:57,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:57,436 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:57,436 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 22:53:57,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:53:57,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:57,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:53:57,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:57,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:53:57,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:57,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:57,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:57,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:57,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:57,444 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:57,444 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:57,444 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:57,444 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:57,445 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,445 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,446 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:57,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:57,446 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,447 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,447 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:57,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:57,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:57,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:57,456 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:57,456 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:57,456 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:57,457 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:57,457 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:57,457 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:57,457 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:57,457 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:57,457 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:57,457 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:57,457 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:57,457 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,461 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:57,461 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:57,465 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,465 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,466 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,466 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:57,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63421e2f[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:57,468 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:57,468 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,471 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:57,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:57,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:57,471 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,471 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:57,475 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:57,475 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,475 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,475 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:57,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:57,479 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:57,479 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:57,479 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:57,479 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:57,479 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:57,483 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,483 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,483 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:57,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,486 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/361e5015002d480795de7912c022fd2f 2023-04-18 22:53:57,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,488 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,491 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,491 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,491 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,491 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:57,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:57,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:57,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:57,491 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:57,491 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:57,491 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:57,491 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:57,491 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,492 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,492 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,491 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:57,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:57,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:57,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858497505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:57,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858497507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,592 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:57,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:57,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:57,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:57,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:57,599 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:57,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:57,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:57,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:57,600 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,600 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,601 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:57,601 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,602 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,602 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:57,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:57,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:57,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858497609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:57,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858497613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:57,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:57,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:57,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:57,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:57,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:57,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:57,628 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:57,629 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:57,629 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:57,629 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:57,629 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:57,629 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:57,629 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,629 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:57,629 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:57,630 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,630 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,631 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,631 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:57,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66f4bc12[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:57,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:57,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,636 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,636 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:57,636 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:57,636 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:57,636 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,636 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:57,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:57,639 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:57,638 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,639 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:57,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:57,642 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:57,642 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:57,642 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:57,642 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:57,642 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:57,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:57,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,645 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,648 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,650 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:57,650 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:57,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:57,651 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,651 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,651 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,651 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,651 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:57,651 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,651 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,651 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:57,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:57,652 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:57,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:57,652 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:57,752 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:57,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:57,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:57,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:57,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:57,757 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:57,757 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:57,758 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:57,758 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:57,758 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,758 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,759 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:57,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:57,760 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,760 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,760 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:57,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:57,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:57,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:57,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:57,761 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:57,761 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:57,761 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:57,764 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:57,764 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:57,764 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:57,764 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:57,764 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:57,764 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:57,764 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:57,764 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:57,764 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,764 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:57,764 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:57,766 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,767 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,767 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,767 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:57,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5daae3ac[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:57,768 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:57,769 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,771 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,771 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:57,771 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,771 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:57,771 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:57,771 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:57,775 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:57,775 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,775 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:57,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:57,780 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:57,780 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:57,780 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:57,780 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:57,780 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:57,781 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,782 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:57,781 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,783 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,785 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,799 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:57,799 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:57,799 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:57,799 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:57,799 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:57,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:57,799 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,800 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,800 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,800 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,800 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,800 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:57,800 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:57,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:57,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:57,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:57,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:57,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46068 deadline: 1681858497812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:57,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858497820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,849 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/8c8b58e6a4ee413b91783e38a9171a43 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8c8b58e6a4ee413b91783e38a9171a43 2023-04-18 22:53:57,849 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/c9427866bb88435ea4bb55873ffef52e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c9427866bb88435ea4bb55873ffef52e 2023-04-18 22:53:57,865 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/B of 3d5ba4e7a99f0fcc1148990ad0a52e36 into 8c8b58e6a4ee413b91783e38a9171a43(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:53:57,865 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/A of 3d5ba4e7a99f0fcc1148990ad0a52e36 into c9427866bb88435ea4bb55873ffef52e(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:53:57,865 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:57,865 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/A, priority=13, startTime=1681858437341; duration=0sec 2023-04-18 22:53:57,865 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:53:57,865 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:53:57,865 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:57,866 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/B, priority=13, startTime=1681858437341; duration=0sec 2023-04-18 22:53:57,866 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:53:57,867 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36666 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:53:57,867 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/C is initiating minor compaction (all files) 2023-04-18 22:53:57,867 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/C in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:57,868 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/faf1319830c1433a995430f9ca9e3261, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6c76f07713c14c2abe68e12f67ff21c2, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c03dba2a81604adfbce9276254a1e03f] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=35.8 K 2023-04-18 22:53:57,868 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting faf1319830c1433a995430f9ca9e3261, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681858435043 2023-04-18 22:53:57,869 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 6c76f07713c14c2abe68e12f67ff21c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1681858435420 2023-04-18 22:53:57,869 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting c03dba2a81604adfbce9276254a1e03f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681858436105 2023-04-18 22:53:57,878 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#C#compaction#40 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:53:57,900 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:57,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:57,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:57,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:57,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:57,906 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:57,906 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:57,906 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:57,906 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:57,907 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,907 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,908 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,908 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:57,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:57,908 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,908 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:57,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:57,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:57,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:57,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:57,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:57,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:57,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:57,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:57,910 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:57,910 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:57,910 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:57,910 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:57,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:57,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:57,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:57,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:57,912 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,912 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,912 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,912 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:57,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@da1eff0[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:57,914 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:57,914 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,915 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/91f57482fdcf4c9e8eeffc9f826885e8 2023-04-18 22:53:57,916 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:57,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:57,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:57,916 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,916 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:57,919 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,919 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,919 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:57,919 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:57,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:57,922 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:57,922 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:57,922 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:57,923 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:57,923 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:57,925 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,925 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:57,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,925 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:57,927 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,928 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,931 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,931 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,931 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:57,931 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:57,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:57,931 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:57,931 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:57,932 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,932 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:57,932 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:57,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:57,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:57,931 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:57,932 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:57,932 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:57,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:57,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:57,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:57,936 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/7f744e3edfb941ea857dfe66ef4f689b 2023-04-18 22:53:57,944 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/361e5015002d480795de7912c022fd2f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/361e5015002d480795de7912c022fd2f 2023-04-18 22:53:57,951 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/361e5015002d480795de7912c022fd2f, entries=150, sequenceid=300, filesize=11.9 K 2023-04-18 22:53:57,953 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/91f57482fdcf4c9e8eeffc9f826885e8 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/91f57482fdcf4c9e8eeffc9f826885e8 2023-04-18 22:53:57,960 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/91f57482fdcf4c9e8eeffc9f826885e8, entries=150, sequenceid=300, filesize=11.9 K 2023-04-18 22:53:57,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/7f744e3edfb941ea857dfe66ef4f689b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/7f744e3edfb941ea857dfe66ef4f689b 2023-04-18 22:53:57,967 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/7f744e3edfb941ea857dfe66ef4f689b, entries=150, sequenceid=300, filesize=11.9 K 2023-04-18 22:53:57,968 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 532ms, sequenceid=300, compaction requested=false 2023-04-18 22:53:57,969 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:58,032 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:58,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:58,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:58,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:58,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:58,037 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:58,037 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:58,037 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:58,037 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:58,037 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,038 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:58,039 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,039 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:58,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:58,040 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,040 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:58,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:58,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:58,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:58,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:58,042 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:58,042 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:58,042 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:58,042 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:58,042 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:58,042 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:58,042 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:58,043 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 22:53:58,043 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:53:58,043 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:58,043 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:53:58,043 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:58,043 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:53:58,043 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:58,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:58,119 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:58,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:58,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858498141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:58,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858498144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:58,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858498245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:58,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858498250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,292 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/e58f3495bbbb4133941a477191498b30 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/e58f3495bbbb4133941a477191498b30 2023-04-18 22:53:58,298 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/C of 3d5ba4e7a99f0fcc1148990ad0a52e36 into e58f3495bbbb4133941a477191498b30(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:53:58,299 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:58,299 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/C, priority=13, startTime=1681858437345; duration=0sec 2023-04-18 22:53:58,299 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:53:58,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:58,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858498449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,461 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/f868a7717baa49a6a647d6168e4a0c83 2023-04-18 22:53:58,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:58,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858498457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,481 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/d52a4e1d300943389a67e2d5e4ce64eb 2023-04-18 22:53:58,505 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/959af490ddd143048421f85b8812fbb8 2023-04-18 22:53:58,513 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/f868a7717baa49a6a647d6168e4a0c83 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/f868a7717baa49a6a647d6168e4a0c83 2023-04-18 22:53:58,520 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/f868a7717baa49a6a647d6168e4a0c83, entries=150, sequenceid=321, filesize=11.9 K 2023-04-18 22:53:58,521 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/d52a4e1d300943389a67e2d5e4ce64eb as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d52a4e1d300943389a67e2d5e4ce64eb 2023-04-18 22:53:58,530 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d52a4e1d300943389a67e2d5e4ce64eb, entries=150, sequenceid=321, filesize=11.9 K 2023-04-18 22:53:58,531 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/959af490ddd143048421f85b8812fbb8 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/959af490ddd143048421f85b8812fbb8 2023-04-18 22:53:58,539 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/959af490ddd143048421f85b8812fbb8, entries=150, sequenceid=321, filesize=11.9 K 2023-04-18 22:53:58,541 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 498ms, sequenceid=321, compaction requested=true 2023-04-18 22:53:58,541 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:58,541 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:58,541 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:58,542 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:58,542 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,542 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:58,542 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:58,545 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:58,546 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:58,547 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:58,547 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:58,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:58,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:58,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:58,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e613b0d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:58,558 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:58,558 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:58,559 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:58,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:58,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:58,560 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,560 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:58,562 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,562 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:58,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:58,562 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:58,562 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:58,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:58,565 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:58,566 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:58,566 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:58,566 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:58,566 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:58,567 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,567 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,567 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:58,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:58,570 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:58,572 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:58,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:58,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:58,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:58,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:58,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:58,576 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:58,576 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:58,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:58,575 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:58,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:58,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:58,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:58,676 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:58,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:58,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:58,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:58,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:58,689 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:58,689 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:58,689 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:58,689 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:58,690 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,690 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:58,691 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:58,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:58,692 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:58,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,697 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,697 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:58,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:58,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:58,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:58,697 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:58,697 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:58,697 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:58,698 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:58,698 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:58,698 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:58,698 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:58,698 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 22:53:58,698 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:53:58,698 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:58,698 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:53:58,699 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:58,699 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:53:58,699 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:58,739 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/863777c3ad4942f8a9d1fc5ae1eff0a6 2023-04-18 22:53:58,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:58,764 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:58,790 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/49ea09cf763a4964bf8b8c6bc9000541 2023-04-18 22:53:58,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:58,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46062 deadline: 1681858498808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:58,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858498813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,822 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/98283a2c381a433799ef2584e0a24b7a 2023-04-18 22:53:58,830 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/863777c3ad4942f8a9d1fc5ae1eff0a6 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/863777c3ad4942f8a9d1fc5ae1eff0a6 2023-04-18 22:53:58,838 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/863777c3ad4942f8a9d1fc5ae1eff0a6, entries=150, sequenceid=335, filesize=11.9 K 2023-04-18 22:53:58,839 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/49ea09cf763a4964bf8b8c6bc9000541 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/49ea09cf763a4964bf8b8c6bc9000541 2023-04-18 22:53:58,850 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/49ea09cf763a4964bf8b8c6bc9000541, entries=150, sequenceid=335, filesize=11.9 K 2023-04-18 22:53:58,851 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/98283a2c381a433799ef2584e0a24b7a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/98283a2c381a433799ef2584e0a24b7a 2023-04-18 22:53:58,858 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/98283a2c381a433799ef2584e0a24b7a, entries=150, sequenceid=335, filesize=11.9 K 2023-04-18 22:53:58,859 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 161ms, sequenceid=335, compaction requested=true 2023-04-18 22:53:58,859 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:58,860 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:58,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:58,861 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:58,861 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,862 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:58,862 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:58,864 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,864 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:58,865 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:58,865 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:58,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:58,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3db60cff[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:58,867 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:58,867 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:58,869 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:58,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:58,869 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:58,869 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:58,869 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,869 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:58,869 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:58,872 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:58,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:58,873 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:58,873 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:58,873 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:58,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:58,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:58,877 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:58,877 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:58,877 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:58,877 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:58,877 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:58,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:58,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:58,882 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:58,883 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,892 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,893 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:58,893 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:58,893 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:58,893 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:58,893 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:58,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:58,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:58,893 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:58,893 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:58,893 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:58,893 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:58,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:58,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:58,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:58,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:58,894 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:58,894 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:58,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:58,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:58,918 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 22:53:58,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858438805, max timestamp=1681858438917Num uniques 859; 2023-04-18 22:53:58,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858438805, max timestamp=1681858438917Num uniques 859; 2023-04-18 22:53:58,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:53:58,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:53:58,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858438805, max timestamp=1681858438917Num uniques 859; 2023-04-18 22:53:58,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:53:58,921 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:53:58,921 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:53:58,921 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:53:58,921 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:53:58,921 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:53:58,921 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:53:58,943 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/35a4684006734133830ea4b440630053 2023-04-18 22:53:58,968 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/d928f41816fd4653a1cdf547137b25ef 2023-04-18 22:53:58,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858438958, max timestamp=1681858438969Num uniques 859; 2023-04-18 22:53:58,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858438958, max timestamp=1681858438969Num uniques 859; 2023-04-18 22:53:58,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:53:58,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858438958, max timestamp=1681858438969Num uniques 859; 2023-04-18 22:53:58,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:53:58,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:53:58,994 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:58,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:58,996 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/fd7435c0747340d4bb054cf32e5def09 2023-04-18 22:53:58,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:59,004 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/35a4684006734133830ea4b440630053 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/35a4684006734133830ea4b440630053 2023-04-18 22:53:59,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:59,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:59,008 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:59,008 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:59,008 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:59,008 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:59,009 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,009 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,010 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,010 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:59,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,011 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,011 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:59,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,012 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/35a4684006734133830ea4b440630053, entries=150, sequenceid=356, filesize=11.9 K 2023-04-18 22:53:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:59,013 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/d928f41816fd4653a1cdf547137b25ef as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d928f41816fd4653a1cdf547137b25ef 2023-04-18 22:53:59,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:59,016 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:59,016 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:59,016 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:59,016 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:59,016 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:59,016 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:59,016 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:59,016 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:59,016 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:59,016 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:59,017 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:59,018 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,018 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:59,019 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,019 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,020 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d928f41816fd4653a1cdf547137b25ef, entries=150, sequenceid=356, filesize=11.9 K 2023-04-18 22:53:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:59,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:59,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@378b2f66[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:59,021 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:59,021 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/fd7435c0747340d4bb054cf32e5def09 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fd7435c0747340d4bb054cf32e5def09 2023-04-18 22:53:59,021 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,028 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,028 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fd7435c0747340d4bb054cf32e5def09, entries=150, sequenceid=356, filesize=11.9 K 2023-04-18 22:53:59,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,029 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:59,029 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:59,029 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:59,029 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,029 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:59,031 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:59,031 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=127.47 KB/130530 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 113ms, sequenceid=356, compaction requested=true 2023-04-18 22:53:59,031 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:59,031 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:59,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:59,031 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:53:59,031 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 22:53:59,031 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 22:53:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:59,031 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:59,032 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:59,032 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:53:59,032 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:53:59,032 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-18 22:53:59,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:59,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:53:59,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:53:59,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:53:59,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:53:59,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:53:59,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:53:59,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,043 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61340 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 22:53:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:59,043 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/B is initiating minor compaction (all files) 2023-04-18 22:53:59,043 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/B in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:59,043 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8c8b58e6a4ee413b91783e38a9171a43, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/91f57482fdcf4c9e8eeffc9f826885e8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d52a4e1d300943389a67e2d5e4ce64eb, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/49ea09cf763a4964bf8b8c6bc9000541, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d928f41816fd4653a1cdf547137b25ef] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=59.9 K 2023-04-18 22:53:59,043 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61340 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 22:53:59,044 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/A is initiating minor compaction (all files) 2023-04-18 22:53:59,044 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/A in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:59,044 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 8c8b58e6a4ee413b91783e38a9171a43, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681858436105 2023-04-18 22:53:59,045 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c9427866bb88435ea4bb55873ffef52e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/361e5015002d480795de7912c022fd2f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/f868a7717baa49a6a647d6168e4a0c83, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/863777c3ad4942f8a9d1fc5ae1eff0a6, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/35a4684006734133830ea4b440630053] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=59.9 K 2023-04-18 22:53:59,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,046 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 91f57482fdcf4c9e8eeffc9f826885e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681858436803 2023-04-18 22:53:59,047 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting c9427866bb88435ea4bb55873ffef52e, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681858436105 2023-04-18 22:53:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,053 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 361e5015002d480795de7912c022fd2f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681858436803 2023-04-18 22:53:59,054 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting d52a4e1d300943389a67e2d5e4ce64eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681858437475 2023-04-18 22:53:59,054 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting f868a7717baa49a6a647d6168e4a0c83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681858437475 2023-04-18 22:53:59,054 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 49ea09cf763a4964bf8b8c6bc9000541, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681858438134 2023-04-18 22:53:59,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:59,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:59,055 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:59,055 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:59,055 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:59,055 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:59,055 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:59,055 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting d928f41816fd4653a1cdf547137b25ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681858438805 2023-04-18 22:53:59,056 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 863777c3ad4942f8a9d1fc5ae1eff0a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681858438134 2023-04-18 22:53:59,056 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 35a4684006734133830ea4b440630053, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681858438805 2023-04-18 22:53:59,058 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,058 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:59,058 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:59,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:59,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:59,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:59,059 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,062 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:59,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,065 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,065 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,065 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,066 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,066 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,066 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,066 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,066 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:59,066 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:59,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:59,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:59,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:59,066 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:59,066 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:59,066 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:59,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:59,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:59,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:59,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858499073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:59,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858499074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,088 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/36590d18f151415cb7ceb2f821a480c3 2023-04-18 22:53:59,090 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#A#compaction#52 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:53:59,090 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#B#compaction#53 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:53:59,132 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/c1470a4367054114b6c0c319ddf12e02 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c1470a4367054114b6c0c319ddf12e02 2023-04-18 22:53:59,133 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/de7167cfe6a04bd68f2fc2651191b4ff as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/de7167cfe6a04bd68f2fc2651191b4ff 2023-04-18 22:53:59,146 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/A of 3d5ba4e7a99f0fcc1148990ad0a52e36 into c1470a4367054114b6c0c319ddf12e02(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:53:59,146 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:59,146 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/A, priority=11, startTime=1681858439031; duration=0sec 2023-04-18 22:53:59,146 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:53:59,147 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 22:53:59,148 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/B of 3d5ba4e7a99f0fcc1148990ad0a52e36 into de7167cfe6a04bd68f2fc2651191b4ff(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:53:59,148 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:59,148 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/B, priority=11, startTime=1681858439031; duration=0sec 2023-04-18 22:53:59,148 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:53:59,149 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61340 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 22:53:59,149 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/C is initiating minor compaction (all files) 2023-04-18 22:53:59,149 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/C in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:59,149 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/e58f3495bbbb4133941a477191498b30, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/7f744e3edfb941ea857dfe66ef4f689b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/959af490ddd143048421f85b8812fbb8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/98283a2c381a433799ef2584e0a24b7a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fd7435c0747340d4bb054cf32e5def09] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=59.9 K 2023-04-18 22:53:59,150 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting e58f3495bbbb4133941a477191498b30, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681858436105 2023-04-18 22:53:59,151 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 7f744e3edfb941ea857dfe66ef4f689b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681858436803 2023-04-18 22:53:59,151 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 959af490ddd143048421f85b8812fbb8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681858437475 2023-04-18 22:53:59,151 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 98283a2c381a433799ef2584e0a24b7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681858438134 2023-04-18 22:53:59,152 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting fd7435c0747340d4bb054cf32e5def09, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681858438805 2023-04-18 22:53:59,163 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#C#compaction#55 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:53:59,166 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:59,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:59,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:59,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:59,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:59,179 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:59,179 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:59,179 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:59,179 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:59,180 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,180 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,182 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:59,182 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:59,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:59,201 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,201 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:59,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858499180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:59,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:59,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:59,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858499201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:59,208 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/56e3b2cd11c54a37ba100c2b25fcfb84 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/56e3b2cd11c54a37ba100c2b25fcfb84 2023-04-18 22:53:59,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858499206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:59,208 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:59,208 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:59,208 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:59,212 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:59,212 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:59,212 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:59,212 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:59,214 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:53:59,214 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:59,214 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:59,215 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:59,216 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:59,217 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,217 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:59,217 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:59,218 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/C of 3d5ba4e7a99f0fcc1148990ad0a52e36 into 56e3b2cd11c54a37ba100c2b25fcfb84(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:53:59,218 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:59,218 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/C, priority=11, startTime=1681858439032; duration=0sec 2023-04-18 22:53:59,219 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,219 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:59,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:59,219 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:53:59,219 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:59,219 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:59,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:59,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:59,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:59,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35579808[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:59,222 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:59,222 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,223 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,223 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:59,224 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:59,224 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:59,224 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,224 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:59,225 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:59,225 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:59,225 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,225 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:59,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:59,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:59,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:59,228 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:59,228 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:59,228 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:59,228 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:59,228 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:59,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:59,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858499229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,233 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:53:59,233 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:59,233 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,234 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,234 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:59,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:59,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,236 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,239 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,239 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,239 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,239 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:59,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:59,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:53:59,240 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,240 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:59,240 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:59,240 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:59,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:59,240 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:59,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:59,240 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:59,240 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,241 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:59,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858499243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,247 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4188 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:53:59,341 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:59,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:59,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:59,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:59,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:59,347 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:59,347 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:59,347 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:59,347 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:59,347 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,347 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,348 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,349 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:59,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,351 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,352 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:59,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:59,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:59,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:59,353 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:59,353 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:59,353 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:59,353 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:59,355 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,355 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:59,355 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,355 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:59,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:59,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@429974fc[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:59,357 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:59,357 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,362 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,362 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:59,362 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:59,363 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,363 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:59,363 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:59,364 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,364 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:59,364 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:59,364 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:59,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:59,367 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:59,367 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:59,367 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:59,367 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:59,368 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:59,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:59,370 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:59,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,371 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,377 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:59,377 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:59,377 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:59,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:59,377 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,377 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:59,377 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:59,378 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:59,378 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,378 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:59,378 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,378 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,378 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:59,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:59,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858499404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:59,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858499406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,478 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:59,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:59,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:59,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:59,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:59,492 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:59,492 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:59,492 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:59,492 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:59,492 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,493 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,494 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:59,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:59,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:59,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:59,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:59,496 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:59,496 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:59,496 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:59,500 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:59,500 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:59,500 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:59,500 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:59,500 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:59,500 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:59,500 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:53:59,500 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:53:59,500 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,501 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:53:59,501 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:53:59,502 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,502 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:59,503 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,503 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:53:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:59,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:53:59,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f4559a4[Count = 0] remaining members to acquire global barrier 2023-04-18 22:53:59,505 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:53:59,505 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,508 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,508 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:53:59,509 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,509 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:53:59,509 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:53:59,509 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:53:59,511 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,511 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:53:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:59,511 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:59,511 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:53:59,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:53:59,515 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:53:59,515 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:53:59,515 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:53:59,515 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:53:59,516 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:53:59,518 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:53:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:53:59,518 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,518 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:59,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:53:59,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:53:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:53:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:53:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,524 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,528 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:59,528 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:53:59,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:53:59,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:53:59,529 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:53:59,529 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:53:59,528 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:59,528 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:53:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:59,529 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,530 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,530 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,530 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,530 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:53:59,530 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:53:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:53:59,551 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/998446da163845f48d7544da04e64d22 2023-04-18 22:53:59,583 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/b34c035620ac4a528c91acce1167f5f1 2023-04-18 22:53:59,594 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/36590d18f151415cb7ceb2f821a480c3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/36590d18f151415cb7ceb2f821a480c3 2023-04-18 22:53:59,602 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/36590d18f151415cb7ceb2f821a480c3, entries=300, sequenceid=394, filesize=19.0 K 2023-04-18 22:53:59,603 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/998446da163845f48d7544da04e64d22 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/998446da163845f48d7544da04e64d22 2023-04-18 22:53:59,615 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/998446da163845f48d7544da04e64d22, entries=150, sequenceid=394, filesize=11.9 K 2023-04-18 22:53:59,616 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/b34c035620ac4a528c91acce1167f5f1 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/b34c035620ac4a528c91acce1167f5f1 2023-04-18 22:53:59,629 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:53:59,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/b34c035620ac4a528c91acce1167f5f1, entries=150, sequenceid=394, filesize=11.9 K 2023-04-18 22:53:59,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:53:59,631 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 599ms, sequenceid=394, compaction requested=false 2023-04-18 22:53:59,631 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:53:59,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:53:59,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:53:59,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:53:59,636 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:53:59,636 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:59,636 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:53:59,636 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:53:59,637 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,637 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,639 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:53:59,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:53:59,639 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:53:59,639 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,639 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:53:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:53:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:53:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:53:59,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:53:59,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:53:59,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:53:59,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:53:59,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:53:59,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:53:59,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:53:59,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:53:59,652 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:53:59,652 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:53:59,653 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-18 22:53:59,653 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:53:59,653 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:59,653 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:53:59,653 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:59,653 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:53:59,653 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:53:59,667 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/5e4d80031651425c888c14f5be592a2e 2023-04-18 22:53:59,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:53:59,712 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:53:59,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858439734, max timestamp=1681858439748Num uniques 859; 2023-04-18 22:53:59,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858439734, max timestamp=1681858439748Num uniques 859; 2023-04-18 22:53:59,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858439734, max timestamp=1681858439748Num uniques 859; 2023-04-18 22:53:59,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:53:59,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:53:59,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:53:59,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:59,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:59,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858499802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858499799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:59,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858499907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:53:59,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:53:59,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858499908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,104 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/becd2133aac14be593c1aa02a9fca1d9 2023-04-18 22:54:00,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:00,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858500109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:00,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858500114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:00,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858500414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:00,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858500418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,521 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/fe86f138bd27449eaa6d7cab5c74e156 2023-04-18 22:54:00,528 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/5e4d80031651425c888c14f5be592a2e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/5e4d80031651425c888c14f5be592a2e 2023-04-18 22:54:00,535 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/5e4d80031651425c888c14f5be592a2e, entries=150, sequenceid=406, filesize=11.9 K 2023-04-18 22:54:00,540 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/becd2133aac14be593c1aa02a9fca1d9 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/becd2133aac14be593c1aa02a9fca1d9 2023-04-18 22:54:00,547 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/becd2133aac14be593c1aa02a9fca1d9, entries=150, sequenceid=406, filesize=11.9 K 2023-04-18 22:54:00,548 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/fe86f138bd27449eaa6d7cab5c74e156 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fe86f138bd27449eaa6d7cab5c74e156 2023-04-18 22:54:00,555 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fe86f138bd27449eaa6d7cab5c74e156, entries=150, sequenceid=406, filesize=11.9 K 2023-04-18 22:54:00,557 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 904ms, sequenceid=406, compaction requested=true 2023-04-18 22:54:00,557 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:00,557 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:00,557 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,561 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:00,561 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:00,568 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,568 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:00,568 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:00,569 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:00,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:00,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21dbbf35[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:00,571 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:00,571 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:00,572 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:00,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:00,572 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:00,572 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:00,573 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:00,573 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,573 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:00,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,575 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:00,575 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:00,576 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:00,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:00,579 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:00,579 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:00,579 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:00,579 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:00,579 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:00,582 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:00,582 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:00,582 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:00,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:00,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:00,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:00,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:00,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:00,583 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:00,585 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,587 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:00,587 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,587 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:00,587 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:00,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:00,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:00,587 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:00,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:00,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:00,588 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:00,588 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,588 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:00,588 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:00,588 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:00,588 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:00,588 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:00,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:00,688 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:00,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:00,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:00,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:00,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:00,700 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:00,700 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:00,700 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:00,700 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:00,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:00,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:00,702 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:00,702 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:00,702 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,702 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:00,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:00,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:00,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:00,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:00,704 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:00,704 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:00,704 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:00,716 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:00,716 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:00,716 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:00,716 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:00,716 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-18 22:54:00,717 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:54:00,717 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:00,717 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:54:00,717 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:00,717 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:54:00,717 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:00,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:00,926 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:00,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:00,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:00,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858500938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:00,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858500937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:01,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858501044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:01,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858501044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,151 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/066b7985127a4933b85c0632de1745cd 2023-04-18 22:54:01,163 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/5ac64f7b2fd1482497cc79fa31c90227 2023-04-18 22:54:01,180 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/c0f6680889c140b9b7d75ad6536881e0 2023-04-18 22:54:01,186 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/066b7985127a4933b85c0632de1745cd as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/066b7985127a4933b85c0632de1745cd 2023-04-18 22:54:01,192 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/066b7985127a4933b85c0632de1745cd, entries=150, sequenceid=443, filesize=11.9 K 2023-04-18 22:54:01,194 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/5ac64f7b2fd1482497cc79fa31c90227 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/5ac64f7b2fd1482497cc79fa31c90227 2023-04-18 22:54:01,200 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/5ac64f7b2fd1482497cc79fa31c90227, entries=150, sequenceid=443, filesize=11.9 K 2023-04-18 22:54:01,201 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/c0f6680889c140b9b7d75ad6536881e0 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c0f6680889c140b9b7d75ad6536881e0 2023-04-18 22:54:01,206 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c0f6680889c140b9b7d75ad6536881e0, entries=150, sequenceid=443, filesize=11.9 K 2023-04-18 22:54:01,207 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 491ms, sequenceid=443, compaction requested=true 2023-04-18 22:54:01,208 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:01,208 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:01,208 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:01,208 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:01,208 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,208 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:01,208 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:01,219 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,219 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,220 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,220 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:01,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28c5f4de[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:01,222 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:01,222 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,240 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:01,241 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:01,241 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,241 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:01,248 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,248 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:01,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:01,253 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:01,253 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:01,253 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:01,253 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:01,253 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:01,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:01,253 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:54:01,255 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:54:01,255 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:01,255 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:54:01,255 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:01,255 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:54:01,255 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:01,257 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,257 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,257 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:01,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,261 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,262 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,266 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,266 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,266 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,266 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:01,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:01,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:01,266 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,267 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,267 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,267 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,267 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:01,267 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:01,266 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:01,268 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:01,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:01,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:01,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:01,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:01,283 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/578796de2ebb436d93ed97d8b86936f5 2023-04-18 22:54:01,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681858441280, max timestamp=1681858441308Num uniques 859; 2023-04-18 22:54:01,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681858441280, max timestamp=1681858441308Num uniques 859; 2023-04-18 22:54:01,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681858441280, max timestamp=1681858441308Num uniques 859; 2023-04-18 22:54:01,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 22:54:01,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 22:54:01,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 22:54:01,313 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/f25263e0c06e4b73bf4e310022bdc8e5 2023-04-18 22:54:01,363 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/6c17f4ad5dfb47e8ab5d1b7f796d8864 2023-04-18 22:54:01,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:01,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858501356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,367 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:01,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:01,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858501363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:01,370 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/578796de2ebb436d93ed97d8b86936f5 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/578796de2ebb436d93ed97d8b86936f5 2023-04-18 22:54:01,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:01,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:01,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:01,374 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:01,374 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:01,374 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:01,374 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:01,374 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,374 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,378 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/578796de2ebb436d93ed97d8b86936f5, entries=200, sequenceid=454, filesize=14.3 K 2023-04-18 22:54:01,379 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/f25263e0c06e4b73bf4e310022bdc8e5 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/f25263e0c06e4b73bf4e310022bdc8e5 2023-04-18 22:54:01,380 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:01,380 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:01,380 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,381 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:01,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:01,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:01,385 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/f25263e0c06e4b73bf4e310022bdc8e5, entries=150, sequenceid=454, filesize=11.9 K 2023-04-18 22:54:01,386 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/6c17f4ad5dfb47e8ab5d1b7f796d8864 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6c17f4ad5dfb47e8ab5d1b7f796d8864 2023-04-18 22:54:01,388 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:01,388 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:01,388 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:01,388 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:01,388 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:01,388 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:01,388 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:01,388 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:01,388 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,391 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6c17f4ad5dfb47e8ab5d1b7f796d8864, entries=150, sequenceid=454, filesize=11.9 K 2023-04-18 22:54:01,392 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 139ms, sequenceid=454, compaction requested=true 2023-04-18 22:54:01,393 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:01,393 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:01,393 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 22:54:01,393 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 22:54:01,393 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:01,393 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:01,395 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61510 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 22:54:01,395 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 71270 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 22:54:01,395 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/B is initiating minor compaction (all files) 2023-04-18 22:54:01,395 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/A is initiating minor compaction (all files) 2023-04-18 22:54:01,395 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/B in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:01,395 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/A in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:01,395 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/de7167cfe6a04bd68f2fc2651191b4ff, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/998446da163845f48d7544da04e64d22, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/becd2133aac14be593c1aa02a9fca1d9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/5ac64f7b2fd1482497cc79fa31c90227, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/f25263e0c06e4b73bf4e310022bdc8e5] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=60.1 K 2023-04-18 22:54:01,395 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c1470a4367054114b6c0c319ddf12e02, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/36590d18f151415cb7ceb2f821a480c3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/5e4d80031651425c888c14f5be592a2e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/066b7985127a4933b85c0632de1745cd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/578796de2ebb436d93ed97d8b86936f5] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=69.6 K 2023-04-18 22:54:01,396 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting de7167cfe6a04bd68f2fc2651191b4ff, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681858438805 2023-04-18 22:54:01,396 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting c1470a4367054114b6c0c319ddf12e02, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681858438805 2023-04-18 22:54:01,396 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:01,396 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:01,396 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 998446da163845f48d7544da04e64d22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1681858439022 2023-04-18 22:54:01,396 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 36590d18f151415cb7ceb2f821a480c3, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1681858439008 2023-04-18 22:54:01,396 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting becd2133aac14be593c1aa02a9fca1d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1681858439065 2023-04-18 22:54:01,397 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 5e4d80031651425c888c14f5be592a2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1681858439065 2023-04-18 22:54:01,397 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 5ac64f7b2fd1482497cc79fa31c90227, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681858439779 2023-04-18 22:54:01,397 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 066b7985127a4933b85c0632de1745cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681858439779 2023-04-18 22:54:01,397 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting f25263e0c06e4b73bf4e310022bdc8e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681858440937 2023-04-18 22:54:01,397 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 578796de2ebb436d93ed97d8b86936f5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681858440931 2023-04-18 22:54:01,398 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,397 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,398 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,398 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:01,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21724bae[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:01,400 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:01,401 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,402 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,402 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:01,402 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:01,402 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:01,406 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,406 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:01,408 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,408 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:01,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,408 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:01,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,408 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:01,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,411 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#B#compaction#66 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:01,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:01,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:01,412 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:01,412 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:01,412 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:01,412 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:01,412 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:01,412 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#A#compaction#67 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:01,415 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,415 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:01,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,415 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:01,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:01,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,416 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,418 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,420 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:01,420 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:01,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:01,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:01,421 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:01,421 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:01,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:01,421 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:01,421 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:01,421 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:01,421 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,421 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,422 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,422 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,422 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,449 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/b6fe8211ad87473a8f017f2eef43e5a8 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b6fe8211ad87473a8f017f2eef43e5a8 2023-04-18 22:54:01,458 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/B of 3d5ba4e7a99f0fcc1148990ad0a52e36 into b6fe8211ad87473a8f017f2eef43e5a8(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:01,458 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:01,458 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/B, priority=11, startTime=1681858441393; duration=0sec 2023-04-18 22:54:01,458 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:01,458 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 22:54:01,460 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61510 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 22:54:01,460 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/C is initiating minor compaction (all files) 2023-04-18 22:54:01,460 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/C in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:01,460 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/56e3b2cd11c54a37ba100c2b25fcfb84, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/b34c035620ac4a528c91acce1167f5f1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fe86f138bd27449eaa6d7cab5c74e156, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c0f6680889c140b9b7d75ad6536881e0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6c17f4ad5dfb47e8ab5d1b7f796d8864] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=60.1 K 2023-04-18 22:54:01,461 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 56e3b2cd11c54a37ba100c2b25fcfb84, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681858438805 2023-04-18 22:54:01,461 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting b34c035620ac4a528c91acce1167f5f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1681858439022 2023-04-18 22:54:01,462 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting fe86f138bd27449eaa6d7cab5c74e156, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1681858439065 2023-04-18 22:54:01,462 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting c0f6680889c140b9b7d75ad6536881e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681858439779 2023-04-18 22:54:01,463 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 6c17f4ad5dfb47e8ab5d1b7f796d8864, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681858440937 2023-04-18 22:54:01,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:01,470 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-18 22:54:01,470 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:54:01,470 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:01,470 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:54:01,470 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:01,470 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:54:01,470 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:01,484 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#C#compaction#69 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:01,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:01,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858501495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:01,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858501495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,499 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/8cb024b05ece434d86ec6b24450ea2fe as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/8cb024b05ece434d86ec6b24450ea2fe 2023-04-18 22:54:01,507 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/C of 3d5ba4e7a99f0fcc1148990ad0a52e36 into 8cb024b05ece434d86ec6b24450ea2fe(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:01,507 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:01,507 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/C, priority=11, startTime=1681858441393; duration=0sec 2023-04-18 22:54:01,507 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:01,521 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:01,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:01,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:01,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:01,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:01,525 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:01,525 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:01,525 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:01,525 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:01,526 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,526 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,527 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:01,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:01,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:01,527 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,528 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,528 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:01,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:01,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:01,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:01,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:01,529 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:01,529 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:01,529 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:01,529 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:01,529 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:01,530 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:01,530 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:01,530 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:01,530 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:01,530 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:01,530 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:01,530 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,530 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:01,530 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:01,531 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,531 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,532 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,532 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:01,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@449fbe5d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:01,534 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:01,534 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,536 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,536 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:01,536 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:01,536 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:01,537 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,537 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:01,538 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,538 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,538 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:01,538 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:01,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:01,540 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:01,540 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:01,540 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:01,541 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:01,541 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:01,542 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,542 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,542 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:01,543 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,544 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,547 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:01,547 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,547 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:01,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:01,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:01,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:01,547 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,547 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,547 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,547 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,547 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,547 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,547 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:01,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:01,547 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:01,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:01,547 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:01,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:01,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:01,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858501599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:01,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858501600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,648 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:01,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:01,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:01,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:01,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:01,662 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:01,662 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:01,662 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:01,662 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:01,663 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,663 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,665 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:01,665 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:01,666 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,666 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:01,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:01,667 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:01,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:01,668 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:01,668 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:01,668 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:01,668 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:01,668 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:01,668 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:01,668 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:01,668 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:01,669 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:01,669 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:01,669 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:01,669 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,669 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:01,669 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:01,670 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,670 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,671 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,671 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:01,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@384183ec[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:01,673 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:01,674 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,675 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,675 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:01,675 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:01,675 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,675 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:01,675 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:01,676 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:01,677 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:01,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:01,680 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:01,680 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:01,680 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:01,680 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:01,680 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:01,688 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,688 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:01,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,688 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,689 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,692 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,695 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:01,695 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:01,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:01,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:01,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,697 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:01,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:01,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:01,698 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:01,698 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:01,799 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:01,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:01,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:01,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:01,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:01,805 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:01,805 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:01,805 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:01,805 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:01,805 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,805 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:01,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858501804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:01,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858501805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,807 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:01,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:01,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:01,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:01,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:01,815 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:01,815 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:01,815 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:01,815 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:01,815 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:01,815 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:01,815 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:01,815 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:01,815 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:01,815 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:01,815 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:01,815 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,816 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:01,816 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:01,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,818 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:01,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fa5f33e[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:01,821 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:01,822 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,827 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,827 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:01,827 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:01,827 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:01,827 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,827 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:01,829 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,829 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:01,829 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:01,829 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:01,834 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:01,834 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:01,834 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:01,834 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:01,834 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:01,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:01,838 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,838 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:01,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,838 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:01,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:01,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,843 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/cb51048b934f4396b521d97b4988394e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/cb51048b934f4396b521d97b4988394e 2023-04-18 22:54:01,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,844 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,846 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,849 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,849 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,849 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,849 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,849 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,849 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,849 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:01,849 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,850 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:01,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:01,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:01,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:01,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:01,850 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:01,850 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:01,850 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:01,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:01,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:01,851 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/A of 3d5ba4e7a99f0fcc1148990ad0a52e36 into cb51048b934f4396b521d97b4988394e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:01,851 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:01,851 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/A, priority=11, startTime=1681858441393; duration=0sec 2023-04-18 22:54:01,851 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:01,893 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/e04c1edda8054fa0b28b656ab04234ed 2023-04-18 22:54:01,943 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/cc4a9b007965466087509b2bf894965b 2023-04-18 22:54:01,950 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:01,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:01,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:01,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:01,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:01,969 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:01,969 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:01,969 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:01,969 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:01,970 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,970 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,972 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,972 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/6cbcda9d5b904acbb806d0a8474fc48c 2023-04-18 22:54:01,973 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:01,973 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,973 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:01,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:01,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:01,974 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:01,975 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:01,975 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:01,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:01,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:01,976 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:01,976 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:01,976 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:01,976 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:01,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:01,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:01,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:01,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:01,979 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/e04c1edda8054fa0b28b656ab04234ed as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/e04c1edda8054fa0b28b656ab04234ed 2023-04-18 22:54:01,979 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,979 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,980 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,980 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:01,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c39e05f[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:01,982 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:01,982 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,983 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,983 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:01,983 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,983 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:01,983 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:01,983 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:01,984 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,984 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:01,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,985 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:01,985 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:01,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:01,987 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:01,988 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:01,988 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:01,988 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:01,988 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:01,989 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/e04c1edda8054fa0b28b656ab04234ed, entries=200, sequenceid=490, filesize=14.3 K 2023-04-18 22:54:01,990 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,990 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/cc4a9b007965466087509b2bf894965b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/cc4a9b007965466087509b2bf894965b 2023-04-18 22:54:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:01,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,990 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,991 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:01,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:01,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:01,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,995 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,997 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:01,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:01,997 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:01,997 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,998 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:01,998 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:01,998 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,998 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:01,998 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:01,998 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:01,998 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:02,000 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:02,000 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:02,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:02,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:02,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:02,012 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/cc4a9b007965466087509b2bf894965b, entries=150, sequenceid=490, filesize=11.9 K 2023-04-18 22:54:02,015 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/6cbcda9d5b904acbb806d0a8474fc48c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6cbcda9d5b904acbb806d0a8474fc48c 2023-04-18 22:54:02,022 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6cbcda9d5b904acbb806d0a8474fc48c, entries=150, sequenceid=490, filesize=11.9 K 2023-04-18 22:54:02,023 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 554ms, sequenceid=490, compaction requested=false 2023-04-18 22:54:02,023 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:02,098 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:02,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:02,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:02,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:02,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:02,108 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:02,108 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:02,108 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:02,108 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:02,111 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,111 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:02,113 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:02,113 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-18 22:54:02,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:02,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:02,114 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,114 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,114 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:02,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,115 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:54:02,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:02,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,115 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:02,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:02,115 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:54:02,115 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:02,115 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:54:02,115 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:02,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:02,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:02,115 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:02,115 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:02,115 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:02,116 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:02,116 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:02,116 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:02,116 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:02,119 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,119 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,120 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:02,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:02,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:02,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:02,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:02,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35edcad3[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:02,128 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:02,128 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,135 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,135 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:02,135 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:02,135 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:02,135 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,135 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:02,137 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:02,137 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:02,137 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:02,137 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:02,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:02,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:02,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:02,146 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:02,146 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:02,146 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:02,146 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:02,146 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:02,149 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,150 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:02,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,150 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:02,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,152 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:02,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:02,154 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:02,157 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,157 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:02,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:02,157 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:02,157 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:02,157 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:02,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858502153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:02,157 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:02,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858502153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:02,157 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,158 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,158 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,159 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,159 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:02,158 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:02,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:02,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:02,182 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/e2a275285cf84dce950747f5fc6ee7c7 2023-04-18 22:54:02,257 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:02,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:02,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:02,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:02,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858502259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:02,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858502260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:02,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:02,264 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:02,264 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:02,264 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:02,264 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:02,265 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,265 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,267 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,267 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:02,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:02,267 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,267 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:02,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:02,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:02,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:02,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:02,269 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:02,269 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:02,269 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:02,272 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:02,272 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:02,272 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:02,272 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:02,274 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,274 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:02,275 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,275 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:02,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:02,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1077161[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:02,277 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:02,277 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,278 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,278 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:02,278 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,278 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:02,278 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:02,278 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:02,283 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:02,283 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:02,283 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:02,283 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:02,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:02,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:02,288 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:02,288 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:02,288 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:02,288 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:02,288 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:02,290 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,290 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,290 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:02,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:02,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:02,291 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,293 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,296 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:02,296 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:02,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:02,296 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,296 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,296 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,296 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,296 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,296 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,296 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,297 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:02,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:02,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:02,297 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:02,297 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:02,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:02,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:02,397 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:02,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:02,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:02,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:02,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:02,405 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:02,405 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:02,405 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:02,405 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:02,406 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,406 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,407 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,407 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:02,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:02,408 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,408 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:02,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:02,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:02,411 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:02,411 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:02,411 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:02,411 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:02,411 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:02,411 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:02,411 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:02,412 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:02,412 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:02,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:02,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:02,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:02,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:02,416 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,416 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:02,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:02,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:02,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:02,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:02,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:02,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:02,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@112b5e7c[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:02,419 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:02,419 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,420 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:02,420 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,420 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:02,422 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:02,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:02,422 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:02,422 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:02,422 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:02,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:02,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:02,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:02,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:02,426 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:02,426 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:02,426 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:02,426 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:02,426 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:02,428 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,428 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,428 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:02,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:02,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,429 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,432 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,437 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:02,437 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:02,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:02,437 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:02,437 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:02,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:02,437 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:02,438 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:02,438 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:02,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:02,437 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,438 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,438 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,438 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,438 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:02,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858502463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:02,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858502463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,538 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:02,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:02,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:02,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:02,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:02,555 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:02,555 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:02,555 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:02,555 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:02,555 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,555 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,557 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:02,557 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:02,557 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,557 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:02,559 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:02,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:02,559 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:02,559 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:02,559 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:02,559 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:02,559 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:02,559 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:02,559 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:02,559 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:02,559 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:02,561 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,562 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,562 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,562 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:02,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:02,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52536f1e[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:02,564 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:02,564 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,565 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,565 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:02,565 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:02,565 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,566 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:02,566 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:02,568 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:02,568 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:02,568 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:02,568 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:02,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:02,571 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:02,571 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:02,571 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:02,571 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:02,571 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:02,573 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,573 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:02,573 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:02,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:02,573 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,575 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,577 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,577 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,577 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:02,577 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,577 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:02,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:02,578 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,578 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,578 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:02,578 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,578 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:02,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:02,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:02,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:02,578 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:02,578 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:02,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:02,602 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/c4d26b4953f64f0186d7c20a12e55f88 2023-04-18 22:54:02,615 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/1fe50a3c92a74148a3168163bf379a1f 2023-04-18 22:54:02,623 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/e2a275285cf84dce950747f5fc6ee7c7 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/e2a275285cf84dce950747f5fc6ee7c7 2023-04-18 22:54:02,630 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/e2a275285cf84dce950747f5fc6ee7c7, entries=200, sequenceid=504, filesize=14.3 K 2023-04-18 22:54:02,631 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/c4d26b4953f64f0186d7c20a12e55f88 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/c4d26b4953f64f0186d7c20a12e55f88 2023-04-18 22:54:02,640 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/c4d26b4953f64f0186d7c20a12e55f88, entries=150, sequenceid=504, filesize=11.9 K 2023-04-18 22:54:02,641 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/1fe50a3c92a74148a3168163bf379a1f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1fe50a3c92a74148a3168163bf379a1f 2023-04-18 22:54:02,649 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1fe50a3c92a74148a3168163bf379a1f, entries=150, sequenceid=504, filesize=11.9 K 2023-04-18 22:54:02,650 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 536ms, sequenceid=504, compaction requested=true 2023-04-18 22:54:02,650 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:02,650 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:02,650 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:02,650 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:02,650 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:02,650 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:02,652 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:02,652 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42188 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:02,652 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/B is initiating minor compaction (all files) 2023-04-18 22:54:02,652 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/A is initiating minor compaction (all files) 2023-04-18 22:54:02,652 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/B in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:02,652 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/A in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:02,652 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b6fe8211ad87473a8f017f2eef43e5a8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/cc4a9b007965466087509b2bf894965b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/c4d26b4953f64f0186d7c20a12e55f88] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=36.4 K 2023-04-18 22:54:02,652 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/cb51048b934f4396b521d97b4988394e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/e04c1edda8054fa0b28b656ab04234ed, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/e2a275285cf84dce950747f5fc6ee7c7] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=41.2 K 2023-04-18 22:54:02,653 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting b6fe8211ad87473a8f017f2eef43e5a8, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681858440937 2023-04-18 22:54:02,653 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting cb51048b934f4396b521d97b4988394e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681858440937 2023-04-18 22:54:02,654 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting cc4a9b007965466087509b2bf894965b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1681858441354 2023-04-18 22:54:02,654 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting e04c1edda8054fa0b28b656ab04234ed, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1681858441353 2023-04-18 22:54:02,654 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting c4d26b4953f64f0186d7c20a12e55f88, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1681858441485 2023-04-18 22:54:02,654 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting e2a275285cf84dce950747f5fc6ee7c7, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1681858441485 2023-04-18 22:54:02,666 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#A#compaction#75 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:02,668 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#B#compaction#76 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:02,678 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:02,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:02,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:02,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:02,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:02,687 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:02,687 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:02,687 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:02,687 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:02,688 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,688 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,690 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,690 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:02,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:02,690 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,690 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:02,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:02,695 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/c9b09c48e28c4c4ead7fe11ceeb89034 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c9b09c48e28c4c4ead7fe11ceeb89034 2023-04-18 22:54:02,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:02,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:02,701 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:02,701 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:02,701 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:02,704 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/A of 3d5ba4e7a99f0fcc1148990ad0a52e36 into c9b09c48e28c4c4ead7fe11ceeb89034(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:02,704 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:02,704 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/A, priority=13, startTime=1681858442650; duration=0sec 2023-04-18 22:54:02,704 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:02,704 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:02,706 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:02,706 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:02,706 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/C is initiating minor compaction (all files) 2023-04-18 22:54:02,706 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:02,706 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/C in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:02,706 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:02,707 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/8cb024b05ece434d86ec6b24450ea2fe, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6cbcda9d5b904acbb806d0a8474fc48c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1fe50a3c92a74148a3168163bf379a1f] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=36.4 K 2023-04-18 22:54:02,707 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:02,707 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 22:54:02,707 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:54:02,707 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:02,707 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:54:02,707 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:02,708 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:54:02,708 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:02,708 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 8cb024b05ece434d86ec6b24450ea2fe, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681858440937 2023-04-18 22:54:02,709 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 6cbcda9d5b904acbb806d0a8474fc48c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1681858441354 2023-04-18 22:54:02,709 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 1fe50a3c92a74148a3168163bf379a1f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1681858441485 2023-04-18 22:54:02,727 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#C#compaction#78 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:02,736 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/c48dd8cd5a1a4ca994232c098f362ba8 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/c48dd8cd5a1a4ca994232c098f362ba8 2023-04-18 22:54:02,743 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/B of 3d5ba4e7a99f0fcc1148990ad0a52e36 into c48dd8cd5a1a4ca994232c098f362ba8(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:02,743 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:02,743 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/B, priority=13, startTime=1681858442650; duration=0sec 2023-04-18 22:54:02,744 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:02,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:02,768 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:02,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:02,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858502799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:02,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858502799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:02,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858502902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:02,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:02,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858502904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:03,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858503104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:03,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858503107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,151 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/5d6f3b6b158f4a728e4b6c0c5503be5b 2023-04-18 22:54:03,158 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/5e9590c5c0eb45d7ac0aa4e26302dd6c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/5e9590c5c0eb45d7ac0aa4e26302dd6c 2023-04-18 22:54:03,165 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/C of 3d5ba4e7a99f0fcc1148990ad0a52e36 into 5e9590c5c0eb45d7ac0aa4e26302dd6c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:03,165 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:03,165 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/C, priority=13, startTime=1681858442650; duration=0sec 2023-04-18 22:54:03,165 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:03,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:03,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46024 deadline: 1681858503247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,250 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8195 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:54:03,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:03,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46074 deadline: 1681858503250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,254 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8199 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:54:03,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:03,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46054 deadline: 1681858503258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,263 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8204 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:54:03,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:03,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858503408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:03,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858503411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,570 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/7e8b9ae792c147ed91efdca6ce495dfa 2023-04-18 22:54:03,629 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/6e4a9ee6f561465eb370896b5bd69d8a 2023-04-18 22:54:03,648 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/5d6f3b6b158f4a728e4b6c0c5503be5b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/5d6f3b6b158f4a728e4b6c0c5503be5b 2023-04-18 22:54:03,659 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/5d6f3b6b158f4a728e4b6c0c5503be5b, entries=150, sequenceid=525, filesize=11.9 K 2023-04-18 22:54:03,660 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/7e8b9ae792c147ed91efdca6ce495dfa as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/7e8b9ae792c147ed91efdca6ce495dfa 2023-04-18 22:54:03,690 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/7e8b9ae792c147ed91efdca6ce495dfa, entries=150, sequenceid=525, filesize=11.9 K 2023-04-18 22:54:03,694 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/6e4a9ee6f561465eb370896b5bd69d8a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6e4a9ee6f561465eb370896b5bd69d8a 2023-04-18 22:54:03,702 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6e4a9ee6f561465eb370896b5bd69d8a, entries=150, sequenceid=525, filesize=11.9 K 2023-04-18 22:54:03,706 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 999ms, sequenceid=525, compaction requested=false 2023-04-18 22:54:03,706 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:03,706 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:03,706 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:03,706 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:03,706 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,708 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:03,708 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:03,711 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,711 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:03,711 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:03,711 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:03,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:03,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:03,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:03,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:03,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9b39cd5[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:03,713 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:03,714 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:03,715 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:03,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:03,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:03,715 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:03,715 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:03,715 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:03,715 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,715 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:03,718 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:03,718 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:03,719 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:03,720 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:03,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:03,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:03,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:03,726 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:03,726 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:03,726 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:03,726 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:03,726 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:03,729 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:03,729 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:03,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:03,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:03,730 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:03,731 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,732 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,741 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:03,741 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:03,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:03,741 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,741 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:03,741 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:03,741 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:03,741 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,741 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:03,741 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:03,741 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:03,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:03,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:03,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:03,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:03,742 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:03,742 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:03,842 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:03,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:03,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:03,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:03,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:03,849 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:03,849 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:03,849 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:03,849 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:03,850 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:03,850 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:03,851 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,852 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:03,853 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,853 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:03,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:03,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:03,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:03,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:03,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:03,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:03,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:03,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:03,862 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:03,862 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:03,862 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:03,872 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:03,873 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:03,873 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:03,873 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:03,873 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 22:54:03,873 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:54:03,873 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:03,874 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:54:03,874 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:03,874 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:54:03,874 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:03,911 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/100d4261877b4cd2a9bc9066ad070687 2023-04-18 22:54:03,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:03,922 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:03,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:03,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858503960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:03,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:03,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858503961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:04,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858504064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:04,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858504066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:04,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858504270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:04,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858504271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,367 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/9e4b36e4972c4cf1845dae3f2453717a 2023-04-18 22:54:04,394 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/f622670534bb4b8abba967c11ead68c1 2023-04-18 22:54:04,402 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/100d4261877b4cd2a9bc9066ad070687 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/100d4261877b4cd2a9bc9066ad070687 2023-04-18 22:54:04,407 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/100d4261877b4cd2a9bc9066ad070687, entries=150, sequenceid=540, filesize=11.9 K 2023-04-18 22:54:04,408 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/9e4b36e4972c4cf1845dae3f2453717a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/9e4b36e4972c4cf1845dae3f2453717a 2023-04-18 22:54:04,414 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/9e4b36e4972c4cf1845dae3f2453717a, entries=150, sequenceid=540, filesize=11.9 K 2023-04-18 22:54:04,415 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/f622670534bb4b8abba967c11ead68c1 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f622670534bb4b8abba967c11ead68c1 2023-04-18 22:54:04,420 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f622670534bb4b8abba967c11ead68c1, entries=150, sequenceid=540, filesize=11.9 K 2023-04-18 22:54:04,421 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 548ms, sequenceid=540, compaction requested=true 2023-04-18 22:54:04,422 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:04,422 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:04,422 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:04,422 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:04,422 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,422 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:04,422 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:04,425 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,425 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:04,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:04,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:04,426 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:04,426 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:04,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:04,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:04,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:04,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:04,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:04,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@338578fb[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:04,428 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:04,428 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:04,430 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:04,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:04,430 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:04,430 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:04,430 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:04,430 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,430 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:04,432 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,432 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:04,432 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:04,432 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:04,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:04,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:04,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:04,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:04,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:04,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:04,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:04,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:04,437 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:04,437 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:04,437 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:04,437 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:04,437 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:04,440 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,440 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:04,440 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:04,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:04,440 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:04,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:04,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:04,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:04,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:04,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:04,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:04,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:04,442 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:04,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:04,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:04,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:04,446 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:04,446 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:04,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:04,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:04,446 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,446 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:04,446 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:04,446 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:04,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:04,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:04,446 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,446 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:04,446 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:04,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:04,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:04,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:04,447 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:04,447 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:04,447 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:04,547 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:04,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:04,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:04,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:04,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:04,550 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:04,550 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:04,550 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:04,550 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:04,551 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:04,551 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:04,552 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:04,552 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:04,553 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,553 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:04,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:04,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:04,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:04,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:04,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:04,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:04,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:04,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:04,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:04,554 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:04,554 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:04,554 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 22:54:04,555 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:54:04,555 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:04,555 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:54:04,555 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:04,555 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:54:04,555 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:04,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:04,576 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:04,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:04,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858504591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:04,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858504592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:04,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858504694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:04,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858504696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:04,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858504897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:04,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858504898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:04,972 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/aafdde08168e4d7db03d6310d666db91 2023-04-18 22:54:05,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:05,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858505202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:05,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858505202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,384 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/22a5f588b27a4b198cec4098d5ff1280 2023-04-18 22:54:05,423 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/79cec455204b44818f75f6a3bf2b6655 2023-04-18 22:54:05,429 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/aafdde08168e4d7db03d6310d666db91 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/aafdde08168e4d7db03d6310d666db91 2023-04-18 22:54:05,435 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/aafdde08168e4d7db03d6310d666db91, entries=150, sequenceid=559, filesize=11.9 K 2023-04-18 22:54:05,438 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/22a5f588b27a4b198cec4098d5ff1280 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/22a5f588b27a4b198cec4098d5ff1280 2023-04-18 22:54:05,444 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/22a5f588b27a4b198cec4098d5ff1280, entries=150, sequenceid=559, filesize=11.9 K 2023-04-18 22:54:05,452 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/79cec455204b44818f75f6a3bf2b6655 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/79cec455204b44818f75f6a3bf2b6655 2023-04-18 22:54:05,459 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/79cec455204b44818f75f6a3bf2b6655, entries=150, sequenceid=559, filesize=11.9 K 2023-04-18 22:54:05,460 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 906ms, sequenceid=559, compaction requested=true 2023-04-18 22:54:05,460 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:05,460 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:05,460 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:05,460 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:05,460 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,464 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:05,464 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:05,465 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,465 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:05,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:05,466 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,466 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:05,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:05,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:05,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:05,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:05,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:05,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78961849[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:05,468 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:05,468 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,469 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,469 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:05,469 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:05,469 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:05,469 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,469 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:05,472 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,472 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:05,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:05,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:05,472 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:05,472 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:05,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:05,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:05,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:05,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:05,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:05,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:05,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:05,474 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:05,474 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:05,475 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:05,475 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:05,475 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:05,476 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:05,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:05,477 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,477 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:05,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:05,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:05,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:05,477 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:05,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:05,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:05,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,479 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:05,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:05,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,481 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,481 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:05,481 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:05,482 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,482 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:05,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:05,482 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:05,482 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,482 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,482 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:05,482 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:05,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:05,482 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:05,482 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:05,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:05,582 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:05,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:05,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:05,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:05,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:05,596 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:05,596 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:05,596 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:05,596 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:05,597 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,597 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:05,598 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:05,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:05,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:05,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:05,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,600 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,600 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:05,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:05,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:05,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:05,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:05,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:05,610 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:05,610 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:05,610 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:05,610 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:05,610 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 22:54:05,610 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:54:05,611 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:05,611 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:54:05,611 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:05,611 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:54:05,611 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:05,622 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/47d8f132e90046d99ae5aba2bab14d12 2023-04-18 22:54:05,644 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/127ab31e0e7e4054bd75d197a3d12d15 2023-04-18 22:54:05,660 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/dfa1622f677e44d8a26cadb7b683a32a 2023-04-18 22:54:05,666 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/47d8f132e90046d99ae5aba2bab14d12 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/47d8f132e90046d99ae5aba2bab14d12 2023-04-18 22:54:05,672 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/47d8f132e90046d99ae5aba2bab14d12, entries=150, sequenceid=572, filesize=11.9 K 2023-04-18 22:54:05,673 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/127ab31e0e7e4054bd75d197a3d12d15 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/127ab31e0e7e4054bd75d197a3d12d15 2023-04-18 22:54:05,679 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/127ab31e0e7e4054bd75d197a3d12d15, entries=150, sequenceid=572, filesize=11.9 K 2023-04-18 22:54:05,680 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/dfa1622f677e44d8a26cadb7b683a32a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/dfa1622f677e44d8a26cadb7b683a32a 2023-04-18 22:54:05,691 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/dfa1622f677e44d8a26cadb7b683a32a, entries=150, sequenceid=572, filesize=11.9 K 2023-04-18 22:54:05,692 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 82ms, sequenceid=572, compaction requested=true 2023-04-18 22:54:05,692 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:05,692 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:05,692 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:05,692 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:05,692 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,696 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:05,696 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:05,698 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,698 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,698 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:05,702 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:05,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:05,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:05,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:05,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f4b3da[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:05,706 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:05,706 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,709 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,709 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:05,709 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:05,709 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:05,709 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,709 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:05,711 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,711 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:05,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,711 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:05,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:05,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:05,711 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:05,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:05,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:05,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:05,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:05,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:05,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:05,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:05,713 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:05,713 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:05,713 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:05,714 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:05,714 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:05,715 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,715 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:05,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:05,716 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,716 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,717 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:05,723 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:05,723 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:05,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:05,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:05,724 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,724 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:05,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:05,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:05,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:05,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:05,724 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:05,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:05,724 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:05,724 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,724 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,724 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,724 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,724 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:05,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:05,724 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:05,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:05,735 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:54:05,736 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:54:05,736 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:05,736 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:54:05,736 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:05,736 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:54:05,736 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:05,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:05,775 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/f71261a8418e46818689a4608f480f97 2023-04-18 22:54:05,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858445774, max timestamp=1681858445788Num uniques 859; 2023-04-18 22:54:05,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:05,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858445774, max timestamp=1681858445788Num uniques 859; 2023-04-18 22:54:05,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:05,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858445774, max timestamp=1681858445788Num uniques 859; 2023-04-18 22:54:05,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:05,802 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/120ebb6cd7614e169e3048432bcca4cc 2023-04-18 22:54:05,825 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:05,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:05,827 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/f9f458c8e8724bc996efc6a5a050ac1d 2023-04-18 22:54:05,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:05,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:05,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:05,832 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:05,832 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:05,832 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:05,832 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:05,833 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,833 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:05,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:05,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858505831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,834 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/f71261a8418e46818689a4608f480f97 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/f71261a8418e46818689a4608f480f97 2023-04-18 22:54:05,835 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:05,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:05,836 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:05,836 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,836 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:05,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:05,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:05,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/f71261a8418e46818689a4608f480f97, entries=200, sequenceid=582, filesize=14.3 K 2023-04-18 22:54:05,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:05,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858505833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,840 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/120ebb6cd7614e169e3048432bcca4cc as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/120ebb6cd7614e169e3048432bcca4cc 2023-04-18 22:54:05,846 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/120ebb6cd7614e169e3048432bcca4cc, entries=150, sequenceid=582, filesize=11.9 K 2023-04-18 22:54:05,848 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/f9f458c8e8724bc996efc6a5a050ac1d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f9f458c8e8724bc996efc6a5a050ac1d 2023-04-18 22:54:05,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:05,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:05,852 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:05,852 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:05,852 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:05,852 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:05,853 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:05,853 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:05,853 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:05,853 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:05,853 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:05,853 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:05,853 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:05,853 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,856 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:05,856 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:05,858 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:05,859 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:05,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,861 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:05,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:05,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24b4c405[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:05,861 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:05,861 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,862 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,866 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,867 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:05,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:05,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:05,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:05,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f9f458c8e8724bc996efc6a5a050ac1d, entries=150, sequenceid=582, filesize=11.9 K 2023-04-18 22:54:05,869 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,869 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:05,870 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 135ms, sequenceid=582, compaction requested=true 2023-04-18 22:54:05,870 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:05,870 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:05,870 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-18 22:54:05,870 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:05,870 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:05,870 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:05,870 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-18 22:54:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:05,873 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76408 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-18 22:54:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:05,873 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/A is initiating minor compaction (all files) 2023-04-18 22:54:05,873 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/A in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,874 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c9b09c48e28c4c4ead7fe11ceeb89034, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/5d6f3b6b158f4a728e4b6c0c5503be5b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/100d4261877b4cd2a9bc9066ad070687, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/aafdde08168e4d7db03d6310d666db91, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/47d8f132e90046d99ae5aba2bab14d12, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/f71261a8418e46818689a4608f480f97] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=74.6 K 2023-04-18 22:54:05,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:05,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:05,874 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting c9b09c48e28c4c4ead7fe11ceeb89034, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1681858441485 2023-04-18 22:54:05,874 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:05,874 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:05,874 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:05,874 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:05,874 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:05,875 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 5d6f3b6b158f4a728e4b6c0c5503be5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1681858442137 2023-04-18 22:54:05,875 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 100d4261877b4cd2a9bc9066ad070687, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1681858442776 2023-04-18 22:54:05,876 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting aafdde08168e4d7db03d6310d666db91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1681858443953 2023-04-18 22:54:05,876 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-18 22:54:05,876 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 47d8f132e90046d99ae5aba2bab14d12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681858444587 2023-04-18 22:54:05,877 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting f71261a8418e46818689a4608f480f97, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1681858445726 2023-04-18 22:54:05,877 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,878 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:05,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:05,878 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:05,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,878 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:05,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:05,880 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,881 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73968 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-18 22:54:05,881 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/B is initiating minor compaction (all files) 2023-04-18 22:54:05,881 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/B in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:05,881 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/c48dd8cd5a1a4ca994232c098f362ba8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/7e8b9ae792c147ed91efdca6ce495dfa, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/9e4b36e4972c4cf1845dae3f2453717a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/22a5f588b27a4b198cec4098d5ff1280, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/127ab31e0e7e4054bd75d197a3d12d15, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/120ebb6cd7614e169e3048432bcca4cc] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=72.2 K 2023-04-18 22:54:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:05,883 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting c48dd8cd5a1a4ca994232c098f362ba8, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1681858441485 2023-04-18 22:54:05,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:05,884 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 7e8b9ae792c147ed91efdca6ce495dfa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1681858442137 2023-04-18 22:54:05,885 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 9e4b36e4972c4cf1845dae3f2453717a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1681858442776 2023-04-18 22:54:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,886 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 22a5f588b27a4b198cec4098d5ff1280, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1681858443953 2023-04-18 22:54:05,887 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,887 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:05,887 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:05,887 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 127ab31e0e7e4054bd75d197a3d12d15, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681858444587 2023-04-18 22:54:05,887 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:05,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:05,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:05,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:05,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:05,887 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:05,887 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:05,888 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,888 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 120ebb6cd7614e169e3048432bcca4cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1681858445726 2023-04-18 22:54:05,888 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:05,888 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:05,888 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,888 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:05,888 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:05,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:05,905 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#A#compaction#93 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:05,923 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#B#compaction#94 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:05,927 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/58a68366eed54ed29e5918d7db298337 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/58a68366eed54ed29e5918d7db298337 2023-04-18 22:54:05,935 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/A of 3d5ba4e7a99f0fcc1148990ad0a52e36 into 58a68366eed54ed29e5918d7db298337(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:05,935 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:05,935 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/A, priority=10, startTime=1681858445870; duration=0sec 2023-04-18 22:54:05,935 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:05,935 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-18 22:54:05,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:05,938 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-18 22:54:05,938 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:54:05,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:05,938 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:54:05,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:05,938 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:54:05,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:05,939 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73968 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-18 22:54:05,939 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/C is initiating minor compaction (all files) 2023-04-18 22:54:05,939 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/C in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:05,940 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/5e9590c5c0eb45d7ac0aa4e26302dd6c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6e4a9ee6f561465eb370896b5bd69d8a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f622670534bb4b8abba967c11ead68c1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/79cec455204b44818f75f6a3bf2b6655, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/dfa1622f677e44d8a26cadb7b683a32a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f9f458c8e8724bc996efc6a5a050ac1d] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=72.2 K 2023-04-18 22:54:05,940 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 5e9590c5c0eb45d7ac0aa4e26302dd6c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1681858441485 2023-04-18 22:54:05,942 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 6e4a9ee6f561465eb370896b5bd69d8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1681858442137 2023-04-18 22:54:05,942 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting f622670534bb4b8abba967c11ead68c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1681858442776 2023-04-18 22:54:05,943 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 79cec455204b44818f75f6a3bf2b6655, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1681858443953 2023-04-18 22:54:05,943 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting dfa1622f677e44d8a26cadb7b683a32a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681858444587 2023-04-18 22:54:05,944 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting f9f458c8e8724bc996efc6a5a050ac1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1681858445726 2023-04-18 22:54:05,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:05,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858505951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:05,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858505952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:05,971 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#C#compaction#96 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:05,988 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/11f96da1d10640ea86e698c22e5bc234 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/11f96da1d10640ea86e698c22e5bc234 2023-04-18 22:54:05,988 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:05,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:06,000 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/C of 3d5ba4e7a99f0fcc1148990ad0a52e36 into 11f96da1d10640ea86e698c22e5bc234(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:06,000 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:06,000 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/C, priority=10, startTime=1681858445870; duration=0sec 2023-04-18 22:54:06,001 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:06,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:06,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:06,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:06,013 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:06,013 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:06,013 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:06,013 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:06,014 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,014 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,015 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:06,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,021 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,021 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:06,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:06,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:06,041 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:06,041 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:06,041 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:06,041 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:06,041 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:06,046 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:06,046 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:06,046 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:06,046 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:06,050 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:06,050 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:06,050 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,050 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:06,050 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:06,051 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,052 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:06,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:06,052 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,052 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:06,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:06,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:06,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:06,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:06,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cee32c8[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:06,054 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:06,054 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,055 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,056 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:06,056 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:06,056 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:06,056 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,056 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:06,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:06,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858506056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:06,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858506056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,060 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:06,060 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,060 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:06,060 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:06,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:06,064 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:06,064 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:06,064 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:06,064 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:06,064 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:06,067 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,067 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:06,067 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:06,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:06,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:06,069 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,072 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:06,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:06,075 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:06,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:06,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:06,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:06,076 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:06,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:06,076 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:06,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:06,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:06,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:06,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:06,176 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:06,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:06,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:06,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:06,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:06,180 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:06,180 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:06,180 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:06,180 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:06,181 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,181 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,182 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:06,182 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:06,183 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,183 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:06,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:06,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:06,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:06,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:06,184 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:06,184 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:06,184 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:06,184 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:06,184 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:06,184 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:06,184 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:06,184 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:06,185 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:06,185 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:06,185 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:06,185 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,185 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:06,185 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:06,186 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,186 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:06,186 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,186 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:06,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:06,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f2214c[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:06,189 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:06,189 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,190 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,190 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:06,190 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:06,190 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:06,190 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,190 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:06,193 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,193 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:06,193 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:06,193 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:06,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:06,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:06,197 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:06,197 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:06,197 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:06,197 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:06,197 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:06,199 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,199 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:06,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,199 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:06,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:06,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,201 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:06,202 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,206 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,206 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,206 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,206 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:06,206 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,206 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,206 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,206 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:06,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:06,206 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:06,206 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:06,208 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:06,208 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:06,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:06,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:06,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:06,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:06,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:06,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:06,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858506259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:06,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858506259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,307 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:06,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:06,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:06,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:06,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:06,310 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:06,310 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:06,310 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:06,310 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:06,311 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,311 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,316 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:06,316 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,317 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,317 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:06,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:06,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:06,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:06,318 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:06,318 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:06,318 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:06,318 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:06,318 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:06,318 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:06,318 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:06,318 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:06,318 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:06,318 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:06,318 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:06,318 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,319 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:06,319 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:06,320 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,320 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:06,321 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,321 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:06,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:06,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:06,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:06,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:06,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@608c366b[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:06,322 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:06,323 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,324 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,324 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:06,324 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:06,324 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:06,324 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,325 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:06,326 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:06,326 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,326 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:06,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,327 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:06,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:06,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:06,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:06,331 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:06,331 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:06,331 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:06,331 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:06,331 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:06,333 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,333 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,333 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:06,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:06,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:06,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:06,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:06,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:06,336 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:06,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,337 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:06,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,340 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,340 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,340 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,340 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,340 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,340 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:06,340 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,340 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:06,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:06,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:06,340 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:06,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:06,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:06,341 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:06,341 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:06,340 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:06,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:06,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:06,364 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/3084d00cfd7c49d296b85ac4aff0b7ef as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/3084d00cfd7c49d296b85ac4aff0b7ef 2023-04-18 22:54:06,371 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/B of 3d5ba4e7a99f0fcc1148990ad0a52e36 into 3084d00cfd7c49d296b85ac4aff0b7ef(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:06,371 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/058fd55c1cb245a1aba33af89f92958a 2023-04-18 22:54:06,371 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:06,371 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/B, priority=10, startTime=1681858445870; duration=0sec 2023-04-18 22:54:06,371 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:06,441 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:06,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:06,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:06,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:06,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:06,445 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:06,445 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:06,445 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:06,445 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:06,445 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,445 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,482 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:06,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:06,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:06,482 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,483 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,483 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:06,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:06,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:06,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:06,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:06,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:06,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:06,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:06,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:06,486 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:06,486 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:06,486 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:06,486 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:06,486 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:06,486 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:06,486 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:06,486 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,486 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:06,486 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:06,528 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,528 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:06,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:06,529 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:06,529 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:06,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:06,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:06,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:06,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@621a5293[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:06,531 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:06,531 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,551 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:06,551 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,551 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:06,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:06,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:06,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:06,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858506566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:06,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858506567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,597 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:06,597 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,597 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:06,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:06,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:06,597 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:06,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:06,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:06,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:06,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:06,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:06,600 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:06,600 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:06,600 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:06,600 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:06,600 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:06,636 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,636 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:06,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:06,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,636 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:06,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:06,637 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:06,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,638 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:06,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,682 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,682 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:06,682 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:06,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:06,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:06,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:06,682 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:06,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:06,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:06,682 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,682 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,682 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,682 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,683 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,683 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,683 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:06,683 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:06,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:06,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:06,783 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:06,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:06,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:06,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:06,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:06,786 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:06,786 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:06,786 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:06,786 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:06,787 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,787 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,793 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/db20b9ba4020476b9a112a4af96983db 2023-04-18 22:54:06,828 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:06,828 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,829 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,829 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:06,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:06,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:06,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:06,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:06,830 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:06,830 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:06,830 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:06,830 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:06,830 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:06,830 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:06,830 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:06,830 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:06,830 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:06,830 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:06,830 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:06,830 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,830 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:06,830 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:06,897 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,897 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:06,898 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,898 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:06,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:06,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2faf446a[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:06,900 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:06,900 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,974 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:06,975 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:06,975 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:06,975 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:06,975 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:06,975 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:07,013 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,013 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:07,013 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:07,013 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:07,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:07,016 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:07,016 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:07,016 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:07,016 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:07,016 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:07,051 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,051 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,051 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:07,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:07,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:07,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:07,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:07,052 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:07,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,053 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:07,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:07,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858507073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:07,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858507074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,105 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,105 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,105 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:07,105 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:07,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:07,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:07,105 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:07,105 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,105 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:07,105 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:07,106 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,106 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:07,106 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,106 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:07,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:07,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:07,206 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:07,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:07,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/4fe7952f18884c648f6ca43a2ca88149 2023-04-18 22:54:07,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:07,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:07,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:07,211 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:07,211 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:07,211 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:07,211 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:07,212 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,212 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/058fd55c1cb245a1aba33af89f92958a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/058fd55c1cb245a1aba33af89f92958a 2023-04-18 22:54:07,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/058fd55c1cb245a1aba33af89f92958a, entries=200, sequenceid=620, filesize=14.3 K 2023-04-18 22:54:07,220 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/db20b9ba4020476b9a112a4af96983db as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/db20b9ba4020476b9a112a4af96983db 2023-04-18 22:54:07,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/db20b9ba4020476b9a112a4af96983db, entries=150, sequenceid=620, filesize=11.9 K 2023-04-18 22:54:07,226 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/4fe7952f18884c648f6ca43a2ca88149 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/4fe7952f18884c648f6ca43a2ca88149 2023-04-18 22:54:07,244 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/4fe7952f18884c648f6ca43a2ca88149, entries=150, sequenceid=620, filesize=11.9 K 2023-04-18 22:54:07,245 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 1307ms, sequenceid=620, compaction requested=false 2023-04-18 22:54:07,245 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:07,251 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:07,251 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:07,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:07,252 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,252 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:07,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:07,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:07,253 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:07,253 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:07,254 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:07,254 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:07,254 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-18 22:54:07,254 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:54:07,254 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:07,254 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:54:07,254 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:07,254 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:54:07,255 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:07,266 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/a75b302f1c984dd4b42d3ef46c9cb795 2023-04-18 22:54:07,293 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/93ea721dfedc4e9eaddb81b74037d101 2023-04-18 22:54:07,314 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/1efc59b97077481a97914596573f34d1 2023-04-18 22:54:07,320 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/a75b302f1c984dd4b42d3ef46c9cb795 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/a75b302f1c984dd4b42d3ef46c9cb795 2023-04-18 22:54:07,330 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/a75b302f1c984dd4b42d3ef46c9cb795, entries=100, sequenceid=631, filesize=9.5 K 2023-04-18 22:54:07,331 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/93ea721dfedc4e9eaddb81b74037d101 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/93ea721dfedc4e9eaddb81b74037d101 2023-04-18 22:54:07,336 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/93ea721dfedc4e9eaddb81b74037d101, entries=100, sequenceid=631, filesize=9.5 K 2023-04-18 22:54:07,337 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/1efc59b97077481a97914596573f34d1 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1efc59b97077481a97914596573f34d1 2023-04-18 22:54:07,342 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1efc59b97077481a97914596573f34d1, entries=100, sequenceid=631, filesize=9.5 K 2023-04-18 22:54:07,343 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 89ms, sequenceid=631, compaction requested=true 2023-04-18 22:54:07,343 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:07,343 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:07,343 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:07,343 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:07,343 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,343 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:07,343 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:07,345 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,345 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:07,345 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,345 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:07,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:07,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:07,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:07,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:07,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a32f892[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:07,347 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:07,348 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,351 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,351 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,351 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:07,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,352 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:07,352 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:07,352 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:07,354 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,354 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:07,354 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:07,354 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:07,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:07,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:07,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:07,357 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:07,357 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:07,357 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:07,357 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:07,357 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:07,359 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,359 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,359 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:07,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,360 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:07,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:07,361 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,363 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:07,363 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:07,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:07,363 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:07,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:07,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:07,363 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:07,363 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,364 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,364 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:07,364 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:07,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:07,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:07,364 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:07,364 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,364 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,364 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,464 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:07,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:07,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:07,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:07,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:07,484 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:07,484 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:07,484 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:07,484 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:07,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,486 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:07,486 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:07,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:07,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,487 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,487 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:07,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:07,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:07,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:07,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:07,492 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:07,492 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:07,492 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:07,500 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:07,500 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:07,500 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:07,500 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:07,500 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:07,500 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:07,501 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:07,501 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:07,501 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,501 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:07,501 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:07,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,504 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:07,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:07,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:07,505 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,505 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:07,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:07,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:07,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:07,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b8ca0d1[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:07,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:07,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,508 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:07,508 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,508 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:07,510 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:07,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:07,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:07,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:07,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:07,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:07,513 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:07,513 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:07,513 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:07,513 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:07,514 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:07,516 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:07,516 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,516 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:07,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:07,517 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:07,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,521 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:07,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:07,525 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:07,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:07,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:07,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:07,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:07,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:07,526 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:07,526 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:07,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:07,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:07,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:07,626 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:07,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:07,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:07,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:07,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:07,630 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:07,630 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:07,630 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:07,630 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:07,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,632 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:07,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:07,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:07,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:07,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:07,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:07,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:07,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:07,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:07,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:07,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:07,634 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:07,634 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:07,635 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:07,635 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:07,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:07,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:07,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:07,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:07,636 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,636 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:07,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:07,637 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:07,637 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:07,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:07,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:07,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:07,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b32fdf9[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:07,638 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:07,639 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,640 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,640 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:07,642 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,642 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:07,642 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:07,642 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:07,647 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,647 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:07,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,647 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:07,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:07,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:07,647 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:07,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:07,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:07,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:07,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:07,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:07,650 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:07,650 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:07,650 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:07,650 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:07,650 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:07,652 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,652 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:07,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:07,652 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:07,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:07,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:07,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:07,653 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:07,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,654 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:07,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,656 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,656 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:07,656 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:07,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:07,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:07,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:07,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:07,656 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,657 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,657 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,657 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,657 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,656 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:07,657 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:07,656 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:07,657 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:07,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:07,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:07,757 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:07,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:07,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:07,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:07,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:07,761 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:07,761 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:07,761 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:07,761 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:07,762 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,762 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,763 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:07,763 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:07,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:07,763 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,763 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:07,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:07,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:07,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:07,764 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:07,764 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:07,765 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:07,765 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:07,765 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:07,765 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:07,765 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:07,765 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:07,765 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:07,765 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:07,765 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:07,765 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,765 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:07,765 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:07,767 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,767 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:07,768 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,768 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:07,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:07,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:07,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eeb7ae1[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:07,770 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:07,770 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,771 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,771 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:07,771 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:07,771 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:07,771 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,771 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:07,775 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:07,776 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:07,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:07,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:07,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:07,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:07,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:07,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:07,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:07,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:07,779 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:07,779 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:07,779 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:07,779 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:07,779 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:07,781 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,781 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:07,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:07,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,781 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:07,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:07,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:07,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:07,782 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:07,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,784 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,786 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,786 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:07,786 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:07,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:07,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:07,786 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,786 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:07,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:07,786 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,786 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,786 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,786 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,786 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:07,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:07,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:07,787 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:07,787 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:07,887 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:07,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:07,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:07,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:07,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:07,890 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:07,891 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:07,891 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:07,891 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:07,891 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,891 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,892 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,892 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:07,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:07,893 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,893 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:07,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:07,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:07,894 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:07,894 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:07,894 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:07,894 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:07,894 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:07,894 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:07,894 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:07,895 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:07,895 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:07,895 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:07,895 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:07,895 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,895 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:07,895 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:07,896 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:07,896 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:07,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:07,899 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:07,899 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c0f1c6[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:07,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:07,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:07,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:07,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:07,901 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,901 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:07,902 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,902 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:07,902 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:07,902 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:07,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:07,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:07,905 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:07,905 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:07,905 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:07,905 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:07,905 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:07,907 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,907 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:07,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,907 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:07,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:07,907 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:07,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,909 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:07,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:07,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,911 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,911 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,911 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:07,911 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:07,911 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:07,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:07,911 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:07,911 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,912 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:07,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:07,911 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:07,911 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:07,912 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:07,912 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:07,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:07,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:07,912 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:07,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:07,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:08,012 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:08,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:08,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:08,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:08,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:08,016 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:08,016 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:08,016 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:08,016 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:08,016 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,017 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,019 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:08,019 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:08,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:08,019 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,020 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:08,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:08,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:08,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:08,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:08,021 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:08,021 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:08,021 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:08,021 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:08,021 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:08,021 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,021 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:08,021 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:08,022 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,022 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:08,022 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:08,022 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,022 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:08,022 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:08,023 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,023 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,024 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,024 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:08,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:08,025 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:08,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5de45d47[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:08,025 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,027 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,027 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:08,027 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:08,027 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:08,027 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,027 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:08,028 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,028 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,028 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:08,028 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:08,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:08,031 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:08,031 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:08,031 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:08,031 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:08,031 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:08,034 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,034 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:08,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,034 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,035 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,036 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,039 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,039 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:08,039 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,039 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:08,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:08,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:08,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:08,039 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,039 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,039 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,039 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:08,039 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:08,039 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:08,039 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,039 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:08,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:08,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:08,090 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:54:08,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:54:08,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:08,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:54:08,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:08,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:54:08,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:08,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858448114, max timestamp=1681858448119Num uniques 859; 2023-04-18 22:54:08,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:08,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858448114, max timestamp=1681858448119Num uniques 859; 2023-04-18 22:54:08,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858448114, max timestamp=1681858448119Num uniques 859; 2023-04-18 22:54:08,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:08,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:08,140 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:08,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:08,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:08,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:08,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:08,143 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:08,143 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:08,143 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:08,143 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:08,144 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,144 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,145 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:08,145 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:08,145 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,145 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:08,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:08,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:08,147 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:08,147 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:08,147 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:08,148 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:08,148 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:08,148 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,148 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:08,148 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:08,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:08,148 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858508145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,148 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:08,148 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:08,148 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,149 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:08,149 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:08,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:08,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858508147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,150 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,150 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,151 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,151 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:08,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:08,152 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:08,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3155ae37[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:08,152 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,154 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,154 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:08,154 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:08,154 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:08,155 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,155 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:08,156 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,156 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:08,156 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:08,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,156 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:08,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:08,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:08,159 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:08,159 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:08,160 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:08,160 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:08,160 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:08,161 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,161 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,161 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:08,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:08,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:08,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,162 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,164 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,175 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,175 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,175 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,175 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,175 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:08,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:08,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:08,175 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,175 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:08,175 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:08,175 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,175 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:08,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:08,175 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:08,175 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:08,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:08,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:08,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:08,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858508250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:08,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858508250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,275 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:08,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:08,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:08,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:08,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:08,279 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:08,279 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:08,279 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:08,279 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:08,280 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,280 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,281 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:08,281 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:08,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:08,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,281 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,281 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:08,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:08,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:08,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:08,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:08,282 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:08,282 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:08,282 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:08,283 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:08,283 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:08,283 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,283 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:08,283 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:08,283 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,283 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:08,283 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:08,283 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,283 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:08,283 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:08,285 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:08,286 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:08,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@449cf68e[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:08,286 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,288 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,288 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:08,288 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:08,288 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:08,288 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,288 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:08,290 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,290 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:08,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,290 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:08,290 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:08,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:08,293 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:08,293 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:08,293 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:08,293 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:08,293 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:08,295 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,295 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:08,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,295 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,297 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,298 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,301 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,301 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:08,301 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:08,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:08,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:08,301 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:08,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:08,301 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:08,301 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:08,301 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,301 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:08,302 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,302 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,302 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,302 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:08,402 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:08,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:08,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:08,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:08,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:08,405 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:08,406 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:08,406 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:08,406 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:08,406 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,406 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,407 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,407 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:08,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:08,408 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,408 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:08,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:08,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:08,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:08,409 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:08,409 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:08,409 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:08,409 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:08,409 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:08,410 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,410 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:08,410 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:08,410 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,410 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:08,410 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:08,410 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,410 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:08,410 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:08,412 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,413 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,413 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:08,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30a6711a[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:08,414 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:08,414 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,415 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,415 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,415 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:08,415 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:08,416 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:08,416 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:08,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:08,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:08,417 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:08,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:08,420 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:08,420 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:08,420 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:08,420 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:08,420 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:08,421 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,421 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:08,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,421 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,422 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,423 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,427 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,427 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,427 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,427 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,428 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,428 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,428 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:08,428 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:08,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:08,428 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:08,428 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:08,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:08,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:08,428 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:08,428 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:08,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:08,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:08,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:08,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:08,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858508452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:08,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858508454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,511 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=646 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/0fd4fec072cf43bd9612ad39461b8dc1 2023-04-18 22:54:08,528 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:08,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:08,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:08,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:08,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:08,533 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:08,533 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:08,533 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:08,533 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:08,533 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,533 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,534 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:08,534 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:08,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:08,535 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,535 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:08,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:08,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:08,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:08,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:08,536 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:08,536 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:08,536 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:08,540 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:08,540 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:08,540 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,540 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:08,540 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:08,540 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,540 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:08,540 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:08,540 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,541 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:08,541 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:08,543 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,543 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,548 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,548 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:08,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:08,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f571c2c[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:08,549 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:08,550 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,551 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:08,551 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,551 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:08,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:08,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:08,553 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,553 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:08,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:08,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:08,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:08,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:08,557 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:08,557 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:08,557 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:08,557 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:08,557 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:08,559 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,559 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,559 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:08,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:08,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:08,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,560 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,561 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,566 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,566 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,566 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,566 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,566 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:08,566 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:08,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:08,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:08,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:08,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:08,566 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:08,566 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,566 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,566 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:08,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:08,566 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:08,566 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:08,667 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:08,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:08,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:08,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:08,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:08,670 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:08,670 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:08,670 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:08,670 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:08,670 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,671 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,675 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:08,675 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:08,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:08,675 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,675 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:08,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:08,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:08,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:08,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:08,676 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:08,677 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:08,677 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:08,677 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:08,677 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:08,677 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,677 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:08,677 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:08,677 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,677 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:08,677 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:08,677 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,677 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:08,677 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:08,679 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,679 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,679 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,679 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:08,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:08,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@536f37e7[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:08,681 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:08,681 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,682 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,682 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:08,682 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:08,682 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:08,682 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,682 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:08,683 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,683 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:08,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,683 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:08,684 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:08,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:08,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:08,686 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:08,686 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:08,686 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:08,686 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:08,686 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:08,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:08,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:08,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:08,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,693 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,694 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:08,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:08,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:08,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:08,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,697 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:08,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:08,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:08,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:08,697 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:08,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,697 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:08,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:08,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:08,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:08,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858508754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:08,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858508757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,797 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:08,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:08,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:08,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:08,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:08,801 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:08,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:08,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:08,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:08,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,802 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,803 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:08,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:08,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:08,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:08,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:08,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:08,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:08,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:08,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:08,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:08,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:08,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:08,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:08,804 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,804 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:08,804 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:08,804 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:08,805 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:08,805 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,805 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:08,805 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:08,806 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,806 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,806 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,806 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:08,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:08,808 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:08,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70e48382[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:08,808 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,811 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,812 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:08,812 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,812 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:08,812 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:08,812 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:08,813 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,813 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:08,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,813 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:08,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,813 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:08,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:08,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:08,815 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:08,815 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:08,815 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:08,815 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:08,815 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:08,816 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,817 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,817 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:08,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:08,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:08,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,818 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,819 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,821 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:08,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:08,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:08,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:08,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:08,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:08,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:08,822 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:08,822 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:08,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:08,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:08,922 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:08,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:08,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:08,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:08,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:08,926 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:08,926 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:08,926 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:08,926 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:08,927 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,927 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,929 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:08,929 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:08,930 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,930 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:08,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:08,930 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=646 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/d356e1d83efd4f11ba1f577c14f9fdf6 2023-04-18 22:54:08,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:08,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:08,931 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:08,931 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:08,931 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:08,931 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:08,931 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:08,931 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,931 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:08,931 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:08,931 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,931 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:08,931 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:08,931 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,931 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:08,931 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:08,933 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,933 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,933 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,933 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:08,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c90225[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:08,943 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:08,944 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,945 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,945 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:08,945 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:08,945 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:08,948 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,948 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:08,949 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,949 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:08,949 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:08,949 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:08,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:08,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:08,952 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:08,952 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:08,952 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:08,952 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:08,952 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:08,955 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,955 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:08,955 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,955 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=646 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/9b813d25952046eaad0ac45bc2d966b3 2023-04-18 22:54:08,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:08,956 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:08,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,962 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/0fd4fec072cf43bd9612ad39461b8dc1 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0fd4fec072cf43bd9612ad39461b8dc1 2023-04-18 22:54:08,963 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:08,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:08,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,968 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:08,968 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:08,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:08,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:08,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:08,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:08,968 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:08,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:08,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:08,969 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:08,969 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:08,968 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,968 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:08,970 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:08,970 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:08,970 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,970 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:08,970 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:08,971 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0fd4fec072cf43bd9612ad39461b8dc1, entries=200, sequenceid=646, filesize=14.3 K 2023-04-18 22:54:08,972 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/d356e1d83efd4f11ba1f577c14f9fdf6 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d356e1d83efd4f11ba1f577c14f9fdf6 2023-04-18 22:54:08,978 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d356e1d83efd4f11ba1f577c14f9fdf6, entries=150, sequenceid=646, filesize=11.9 K 2023-04-18 22:54:08,979 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/9b813d25952046eaad0ac45bc2d966b3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/9b813d25952046eaad0ac45bc2d966b3 2023-04-18 22:54:08,984 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/9b813d25952046eaad0ac45bc2d966b3, entries=150, sequenceid=646, filesize=11.9 K 2023-04-18 22:54:08,985 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 895ms, sequenceid=646, compaction requested=true 2023-04-18 22:54:08,985 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:08,985 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:08,985 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:54:08,985 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:54:08,985 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:08,985 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:08,987 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52236 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:54:08,987 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/A is initiating minor compaction (all files) 2023-04-18 22:54:08,987 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47356 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:54:08,987 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/A in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,987 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/B is initiating minor compaction (all files) 2023-04-18 22:54:08,987 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/58a68366eed54ed29e5918d7db298337, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/058fd55c1cb245a1aba33af89f92958a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/a75b302f1c984dd4b42d3ef46c9cb795, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0fd4fec072cf43bd9612ad39461b8dc1] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=51.0 K 2023-04-18 22:54:08,987 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/B in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:08,987 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/3084d00cfd7c49d296b85ac4aff0b7ef, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/db20b9ba4020476b9a112a4af96983db, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/93ea721dfedc4e9eaddb81b74037d101, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d356e1d83efd4f11ba1f577c14f9fdf6] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=46.2 K 2023-04-18 22:54:08,987 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 58a68366eed54ed29e5918d7db298337, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1681858445726 2023-04-18 22:54:08,988 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 3084d00cfd7c49d296b85ac4aff0b7ef, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1681858445726 2023-04-18 22:54:08,988 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 058fd55c1cb245a1aba33af89f92958a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1681858445805 2023-04-18 22:54:08,988 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting db20b9ba4020476b9a112a4af96983db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1681858445805 2023-04-18 22:54:08,989 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting a75b302f1c984dd4b42d3ef46c9cb795, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1681858445948 2023-04-18 22:54:08,989 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 93ea721dfedc4e9eaddb81b74037d101, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1681858445948 2023-04-18 22:54:08,990 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 0fd4fec072cf43bd9612ad39461b8dc1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681858448086 2023-04-18 22:54:08,990 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting d356e1d83efd4f11ba1f577c14f9fdf6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681858448086 2023-04-18 22:54:09,034 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#A#compaction#105 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:09,038 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#B#compaction#106 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:09,069 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:09,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:09,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:09,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:09,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:09,080 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:09,080 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:09,080 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:09,080 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:09,081 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,081 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:09,082 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:09,082 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:09,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:09,083 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:09,083 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:09,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:09,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:09,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:09,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:09,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:09,084 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:09,085 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:09,085 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:09,088 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:09,088 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:09,088 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:09,088 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:09,088 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-18 22:54:09,088 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:54:09,088 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:09,088 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:54:09,089 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:09,089 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:54:09,089 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:09,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:09,261 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:09,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:09,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:09,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858509272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858509272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:09,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:09,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858509379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858509379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,471 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/aa3e78baca574ce6a9cf7803247d92c0 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/aa3e78baca574ce6a9cf7803247d92c0 2023-04-18 22:54:09,477 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/A of 3d5ba4e7a99f0fcc1148990ad0a52e36 into aa3e78baca574ce6a9cf7803247d92c0(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:09,477 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:09,477 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/A, priority=12, startTime=1681858448985; duration=0sec 2023-04-18 22:54:09,477 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:09,477 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:54:09,478 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47356 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:54:09,478 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/C is initiating minor compaction (all files) 2023-04-18 22:54:09,478 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/C in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:09,479 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/11f96da1d10640ea86e698c22e5bc234, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/4fe7952f18884c648f6ca43a2ca88149, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1efc59b97077481a97914596573f34d1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/9b813d25952046eaad0ac45bc2d966b3] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=46.2 K 2023-04-18 22:54:09,479 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 11f96da1d10640ea86e698c22e5bc234, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1681858445726 2023-04-18 22:54:09,479 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 4fe7952f18884c648f6ca43a2ca88149, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1681858445805 2023-04-18 22:54:09,480 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 1efc59b97077481a97914596573f34d1, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1681858445948 2023-04-18 22:54:09,481 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 9b813d25952046eaad0ac45bc2d966b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681858448086 2023-04-18 22:54:09,489 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#C#compaction#108 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:09,500 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/37fc8e6877d043b291c4ebad632c0150 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/37fc8e6877d043b291c4ebad632c0150 2023-04-18 22:54:09,503 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/f74cf5f78f744b4fb638409fe6716e04 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f74cf5f78f744b4fb638409fe6716e04 2023-04-18 22:54:09,506 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/B of 3d5ba4e7a99f0fcc1148990ad0a52e36 into 37fc8e6877d043b291c4ebad632c0150(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:09,506 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:09,506 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/B, priority=12, startTime=1681858448985; duration=0sec 2023-04-18 22:54:09,506 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:09,508 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/C of 3d5ba4e7a99f0fcc1148990ad0a52e36 into f74cf5f78f744b4fb638409fe6716e04(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:09,508 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:09,508 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/C, priority=12, startTime=1681858448985; duration=0sec 2023-04-18 22:54:09,508 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:09,509 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=682 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/c38d189fea0a40a0a1a15182cb34a42c 2023-04-18 22:54:09,520 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=682 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/f300db3469f14ceb97b5b34be993a477 2023-04-18 22:54:09,533 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=682 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/1c41fee9c61e4f2fb9d2e5b2943142ec 2023-04-18 22:54:09,538 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/c38d189fea0a40a0a1a15182cb34a42c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c38d189fea0a40a0a1a15182cb34a42c 2023-04-18 22:54:09,554 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c38d189fea0a40a0a1a15182cb34a42c, entries=150, sequenceid=682, filesize=11.9 K 2023-04-18 22:54:09,555 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/f300db3469f14ceb97b5b34be993a477 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/f300db3469f14ceb97b5b34be993a477 2023-04-18 22:54:09,562 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/f300db3469f14ceb97b5b34be993a477, entries=150, sequenceid=682, filesize=11.9 K 2023-04-18 22:54:09,563 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/1c41fee9c61e4f2fb9d2e5b2943142ec as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1c41fee9c61e4f2fb9d2e5b2943142ec 2023-04-18 22:54:09,568 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1c41fee9c61e4f2fb9d2e5b2943142ec, entries=150, sequenceid=682, filesize=11.9 K 2023-04-18 22:54:09,570 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 482ms, sequenceid=682, compaction requested=false 2023-04-18 22:54:09,570 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:09,570 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:09,570 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:09,570 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:09,570 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,570 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:09,570 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:09,573 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,573 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:09,574 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:09,574 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:09,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:09,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:09,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:09,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@408c60c[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:09,576 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:09,576 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:09,577 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:09,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:09,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:09,577 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:09,577 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:09,577 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:09,577 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,577 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:09,578 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:09,579 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,579 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:09,579 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:09,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:09,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:09,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:09,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:09,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:09,582 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:09,582 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:09,582 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:09,582 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:09,583 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:09,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:09,585 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-18 22:54:09,585 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,585 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:09,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,586 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:09,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:09,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,587 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:09,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:09,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:09,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,588 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:09,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,592 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:09,592 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:09,592 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:09,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:09,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:09,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:09,593 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:09,593 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:09,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:09,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:09,593 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,593 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:09,593 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:09,593 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,593 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:09,593 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:09,593 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:54:09,596 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:09,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:54:09,596 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:09,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:54:09,596 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:09,629 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=697 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/1b2e54a4117a48ea8ea65b842696d750 2023-04-18 22:54:09,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:09,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858509636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:09,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858509637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,647 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=697 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/7a22b83b8e084af1b07bfb255f855f53 2023-04-18 22:54:09,693 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:09,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:09,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:09,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:09,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:09,698 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:09,698 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:09,698 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:09,698 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:09,699 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,699 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:09,700 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,700 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:09,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:09,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:09,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:09,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:09,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:09,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:09,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:09,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:09,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:09,702 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:09,702 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:09,702 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:09,703 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:09,703 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:09,703 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:09,709 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:09,710 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:09,710 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:09,710 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:09,710 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:09,710 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,710 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:09,710 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:09,710 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=697 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/a338f060130348efb85902a0cce3987a 2023-04-18 22:54:09,712 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:09,713 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:09,713 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:09,717 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/1b2e54a4117a48ea8ea65b842696d750 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/1b2e54a4117a48ea8ea65b842696d750 2023-04-18 22:54:09,719 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:09,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:09,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:09,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,723 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/1b2e54a4117a48ea8ea65b842696d750, entries=150, sequenceid=697, filesize=11.9 K 2023-04-18 22:54:09,724 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/7a22b83b8e084af1b07bfb255f855f53 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/7a22b83b8e084af1b07bfb255f855f53 2023-04-18 22:54:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:09,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:09,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e721965[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:09,729 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:09,729 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:09,731 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:09,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:09,731 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:09,731 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:09,731 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:09,731 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/7a22b83b8e084af1b07bfb255f855f53, entries=150, sequenceid=697, filesize=11.9 K 2023-04-18 22:54:09,731 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,732 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:09,733 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/a338f060130348efb85902a0cce3987a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/a338f060130348efb85902a0cce3987a 2023-04-18 22:54:09,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:09,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:09,734 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:09,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:09,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:09,743 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:09,743 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:09,743 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:09,743 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:09,743 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:09,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:09,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858509742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:09,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858509743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,746 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/a338f060130348efb85902a0cce3987a, entries=150, sequenceid=697, filesize=11.9 K 2023-04-18 22:54:09,747 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 161ms, sequenceid=697, compaction requested=true 2023-04-18 22:54:09,747 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:09,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:09,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:09,747 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:09,747 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:09,747 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-18 22:54:09,747 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,747 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:09,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,748 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:09,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:09,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:09,748 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:09,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:09,750 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:09,750 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/A is initiating minor compaction (all files) 2023-04-18 22:54:09,750 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/A in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:09,751 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/aa3e78baca574ce6a9cf7803247d92c0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c38d189fea0a40a0a1a15182cb34a42c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/1b2e54a4117a48ea8ea65b842696d750] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=36.9 K 2023-04-18 22:54:09,751 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:09,751 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/B is initiating minor compaction (all files) 2023-04-18 22:54:09,751 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/B in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:09,751 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/37fc8e6877d043b291c4ebad632c0150, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/f300db3469f14ceb97b5b34be993a477, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/7a22b83b8e084af1b07bfb255f855f53] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=36.9 K 2023-04-18 22:54:09,753 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,753 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting aa3e78baca574ce6a9cf7803247d92c0, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681858448086 2023-04-18 22:54:09,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:09,754 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting c38d189fea0a40a0a1a15182cb34a42c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=682, earliestPutTs=1681858448134 2023-04-18 22:54:09,754 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 37fc8e6877d043b291c4ebad632c0150, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681858448086 2023-04-18 22:54:09,754 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 1b2e54a4117a48ea8ea65b842696d750, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=697, earliestPutTs=1681858449260 2023-04-18 22:54:09,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:09,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,755 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting f300db3469f14ceb97b5b34be993a477, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=682, earliestPutTs=1681858448134 2023-04-18 22:54:09,756 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,756 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:09,756 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 7a22b83b8e084af1b07bfb255f855f53, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=697, earliestPutTs=1681858449260 2023-04-18 22:54:09,756 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:09,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:09,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:09,756 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:09,756 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,756 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:09,756 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:09,756 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,756 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:09,757 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:09,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:09,757 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:09,757 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:09,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:09,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:09,769 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#A#compaction#114 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:09,769 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#B#compaction#115 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:09,809 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/4e094655e9c44911af97a215c79e2a5a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/4e094655e9c44911af97a215c79e2a5a 2023-04-18 22:54:09,816 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/A of 3d5ba4e7a99f0fcc1148990ad0a52e36 into 4e094655e9c44911af97a215c79e2a5a(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:09,816 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:09,816 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/A, priority=13, startTime=1681858449747; duration=0sec 2023-04-18 22:54:09,816 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:09,816 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:09,817 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:09,817 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/C is initiating minor compaction (all files) 2023-04-18 22:54:09,818 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/C in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:09,818 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f74cf5f78f744b4fb638409fe6716e04, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1c41fee9c61e4f2fb9d2e5b2943142ec, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/a338f060130348efb85902a0cce3987a] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=36.9 K 2023-04-18 22:54:09,818 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting f74cf5f78f744b4fb638409fe6716e04, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681858448086 2023-04-18 22:54:09,819 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 1c41fee9c61e4f2fb9d2e5b2943142ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=682, earliestPutTs=1681858448134 2023-04-18 22:54:09,819 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting a338f060130348efb85902a0cce3987a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=697, earliestPutTs=1681858449260 2023-04-18 22:54:09,853 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#C#compaction#116 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:09,857 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:09,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:09,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:09,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:09,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:09,860 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:09,861 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:09,861 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:09,861 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:09,861 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,861 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:09,862 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:09,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:09,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:09,863 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:09,863 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,863 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:09,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:09,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:09,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:09,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:09,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:09,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:09,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:09,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:09,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:09,870 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:09,870 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:09,871 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:09,871 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. started... 2023-04-18 22:54:09,871 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 22:54:09,871 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:54:09,871 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:09,872 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:54:09,872 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:09,872 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:54:09,872 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:09,880 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/b4c4739facad40b3bd782eca671b6235 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/b4c4739facad40b3bd782eca671b6235 2023-04-18 22:54:09,888 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/C of 3d5ba4e7a99f0fcc1148990ad0a52e36 into b4c4739facad40b3bd782eca671b6235(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:09,888 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:09,888 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/C, priority=13, startTime=1681858449747; duration=0sec 2023-04-18 22:54:09,888 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:09,949 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. as already flushing 2023-04-18 22:54:09,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:09,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:09,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858509966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:09,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:09,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858509968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:10,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:10,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858510074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:10,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:10,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858510075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:10,100 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d24bbb7 to 127.0.0.1:50825 2023-04-18 22:54:10,100 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:10,102 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49d71120 to 127.0.0.1:50825 2023-04-18 22:54:10,102 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:10,103 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x60b8c4c7 to 127.0.0.1:50825 2023-04-18 22:54:10,103 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:10,105 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2879e2e1 to 127.0.0.1:50825 2023-04-18 22:54:10,105 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:10,209 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/a5034a66acf144d1b472986fe6cca190 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/a5034a66acf144d1b472986fe6cca190 2023-04-18 22:54:10,215 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/B of 3d5ba4e7a99f0fcc1148990ad0a52e36 into a5034a66acf144d1b472986fe6cca190(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:10,216 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:10,216 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/B, priority=13, startTime=1681858449747; duration=0sec 2023-04-18 22:54:10,216 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:10,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:10,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858510277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:10,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:10,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858510280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:10,300 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/0225c5c863684172a9e93990809fa53f 2023-04-18 22:54:10,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:10,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858510579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:10,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:10,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858510584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:10,714 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/c2958adb0e074baa8e7dc4bbbb63a138 2023-04-18 22:54:11,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:11,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:11,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46068 deadline: 1681858511085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:11,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46062 deadline: 1681858511085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:11,136 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/6daadaec54cd4b61bcf8a50f1da9704f 2023-04-18 22:54:11,141 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/0225c5c863684172a9e93990809fa53f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0225c5c863684172a9e93990809fa53f 2023-04-18 22:54:11,145 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0225c5c863684172a9e93990809fa53f, entries=150, sequenceid=718, filesize=11.9 K 2023-04-18 22:54:11,146 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/c2958adb0e074baa8e7dc4bbbb63a138 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/c2958adb0e074baa8e7dc4bbbb63a138 2023-04-18 22:54:11,151 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/c2958adb0e074baa8e7dc4bbbb63a138, entries=150, sequenceid=718, filesize=11.9 K 2023-04-18 22:54:11,151 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/6daadaec54cd4b61bcf8a50f1da9704f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6daadaec54cd4b61bcf8a50f1da9704f 2023-04-18 22:54:11,156 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6daadaec54cd4b61bcf8a50f1da9704f, entries=150, sequenceid=718, filesize=11.9 K 2023-04-18 22:54:11,157 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 1286ms, sequenceid=718, compaction requested=false 2023-04-18 22:54:11,157 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:11,157 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:11,157 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:11,157 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:11,157 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:11,157 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:11,157 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:11,159 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:11,159 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:11,159 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:11,159 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:11,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:11,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:11,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:11,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:11,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ade67ba[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:11,160 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:11,161 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:11,161 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:11,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:11,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:11,162 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:11,162 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:11,162 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:11,162 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:11,162 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:11,164 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:11,164 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:11,164 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:11,164 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:11,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:11,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:11,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:11,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:11,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:11,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:11,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:11,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:11,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:11,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:11,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:11,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:11,167 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:11,167 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:11,167 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:11,167 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:11,167 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:11,168 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:11,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:11,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:11,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:11,168 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:11,168 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:11,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:11,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:11,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:11,169 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:11,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:11,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:11,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:11,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:11,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:11,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:11,170 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:11,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:11,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:11,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:11,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:11,172 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:11,172 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:11,172 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:11,172 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:11,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:11,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:11,172 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:11,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:11,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:11,172 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:11,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:11,172 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:11,172 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:11,172 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:11,172 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:11,172 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:11,172 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:11,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:11,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:11,272 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:11,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:12,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:12,090 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 22:54:12,090 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30eb7fb2 to 127.0.0.1:50825 2023-04-18 22:54:12,091 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:54:12,091 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:12,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:12,091 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:54:12,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:12,091 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:54:12,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:12,092 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46192d17 to 127.0.0.1:50825 2023-04-18 22:54:12,092 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:12,499 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/6511fb58840d4da7a89895faff0e9e6c 2023-04-18 22:54:12,911 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/85dfd051a8614380ba1feb7945b0f34a 2023-04-18 22:54:13,279 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79eb61dd to 127.0.0.1:50825 2023-04-18 22:54:13,279 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:13,322 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e91d243 to 127.0.0.1:50825 2023-04-18 22:54:13,322 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:13,322 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/147410aaff8d4bb1a6e5864887b5f36d 2023-04-18 22:54:13,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/6511fb58840d4da7a89895faff0e9e6c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/6511fb58840d4da7a89895faff0e9e6c 2023-04-18 22:54:13,331 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/6511fb58840d4da7a89895faff0e9e6c, entries=150, sequenceid=733, filesize=11.9 K 2023-04-18 22:54:13,332 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/85dfd051a8614380ba1feb7945b0f34a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/85dfd051a8614380ba1feb7945b0f34a 2023-04-18 22:54:13,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/85dfd051a8614380ba1feb7945b0f34a, entries=150, sequenceid=733, filesize=11.9 K 2023-04-18 22:54:13,337 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/147410aaff8d4bb1a6e5864887b5f36d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/147410aaff8d4bb1a6e5864887b5f36d 2023-04-18 22:54:13,339 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x748f869b to 127.0.0.1:50825 2023-04-18 22:54:13,340 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:13,340 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-18 22:54:13,340 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 53 2023-04-18 22:54:13,340 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 48 2023-04-18 22:54:13,340 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 56 2023-04-18 22:54:13,340 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 212 2023-04-18 22:54:13,340 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 239 2023-04-18 22:54:13,340 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-18 22:54:13,340 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(398): read 4106 2023-04-18 22:54:13,340 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(398): read 3923 2023-04-18 22:54:13,340 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-18 22:54:13,340 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(402): scanned 1832 2023-04-18 22:54:13,340 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(403): verified 5493 rows 2023-04-18 22:54:13,340 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(402): scanned 1837 2023-04-18 22:54:13,340 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(403): verified 5506 rows 2023-04-18 22:54:13,340 INFO [Listener at localhost/36651] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-18 22:54:13,341 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d9fc901 to 127.0.0.1:50825 2023-04-18 22:54:13,341 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:13,341 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/147410aaff8d4bb1a6e5864887b5f36d, entries=150, sequenceid=733, filesize=11.9 K 2023-04-18 22:54:13,342 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=26.84 KB/27480 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 1252ms, sequenceid=733, compaction requested=true 2023-04-18 22:54:13,342 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:13,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:13,342 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:13,342 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:13,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:13,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:13,344 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:13,344 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/B is initiating minor compaction (all files) 2023-04-18 22:54:13,344 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:13,344 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/B in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:13,344 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 3d5ba4e7a99f0fcc1148990ad0a52e36/A is initiating minor compaction (all files) 2023-04-18 22:54:13,344 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/a5034a66acf144d1b472986fe6cca190, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/c2958adb0e074baa8e7dc4bbbb63a138, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/85dfd051a8614380ba1feb7945b0f34a] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=37.0 K 2023-04-18 22:54:13,344 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d5ba4e7a99f0fcc1148990ad0a52e36/A in TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:13,345 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/4e094655e9c44911af97a215c79e2a5a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0225c5c863684172a9e93990809fa53f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/6511fb58840d4da7a89895faff0e9e6c] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp, totalSize=37.0 K 2023-04-18 22:54:13,345 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting a5034a66acf144d1b472986fe6cca190, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=697, earliestPutTs=1681858449260 2023-04-18 22:54:13,345 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 4e094655e9c44911af97a215c79e2a5a, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=697, earliestPutTs=1681858449260 2023-04-18 22:54:13,345 INFO [Listener at localhost/36651] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-18 22:54:13,346 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting c2958adb0e074baa8e7dc4bbbb63a138, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1681858449631 2023-04-18 22:54:13,346 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 0225c5c863684172a9e93990809fa53f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1681858449631 2023-04-18 22:54:13,346 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 85dfd051a8614380ba1feb7945b0f34a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1681858449949 2023-04-18 22:54:13,346 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 6511fb58840d4da7a89895faff0e9e6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1681858449949 2023-04-18 22:54:13,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-18 22:54:13,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-18 22:54:13,360 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#A#compaction#123 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:13,362 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d5ba4e7a99f0fcc1148990ad0a52e36#B#compaction#124 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:13,363 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858453363"}]},"ts":"1681858453363"} 2023-04-18 22:54:13,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-18 22:54:13,364 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-18 22:54:13,366 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-18 22:54:13,367 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3d5ba4e7a99f0fcc1148990ad0a52e36, UNASSIGN}] 2023-04-18 22:54:13,369 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3d5ba4e7a99f0fcc1148990ad0a52e36, UNASSIGN 2023-04-18 22:54:13,370 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=3d5ba4e7a99f0fcc1148990ad0a52e36, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:13,370 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858453370"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858453370"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858453370"}]},"ts":"1681858453370"} 2023-04-18 22:54:13,372 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:54:13,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-18 22:54:13,531 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:13,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 3d5ba4e7a99f0fcc1148990ad0a52e36, disabling compactions & flushes 2023-04-18 22:54:13,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 2 compactions to complete for region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:13,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-18 22:54:13,771 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/5fd9162949894412b79fb943279c73b2 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/5fd9162949894412b79fb943279c73b2 2023-04-18 22:54:13,775 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/8f676b22859942eda70d98350a0bea3a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8f676b22859942eda70d98350a0bea3a 2023-04-18 22:54:13,777 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/A of 3d5ba4e7a99f0fcc1148990ad0a52e36 into 5fd9162949894412b79fb943279c73b2(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:13,777 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:13,777 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/A, priority=13, startTime=1681858453342; duration=0sec 2023-04-18 22:54:13,777 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:13,777 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. because compaction request was cancelled 2023-04-18 22:54:13,779 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d5ba4e7a99f0fcc1148990ad0a52e36/B of 3d5ba4e7a99f0fcc1148990ad0a52e36 into 8f676b22859942eda70d98350a0bea3a(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:13,780 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:13,780 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:13,780 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36., storeName=3d5ba4e7a99f0fcc1148990ad0a52e36/B, priority=13, startTime=1681858453342; duration=0sec 2023-04-18 22:54:13,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:13,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. after waiting 0 ms 2023-04-18 22:54:13,780 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:13,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:13,780 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 3d5ba4e7a99f0fcc1148990ad0a52e36 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-18 22:54:13,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=A 2023-04-18 22:54:13,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:13,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=B 2023-04-18 22:54:13,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:13,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d5ba4e7a99f0fcc1148990ad0a52e36, store=C 2023-04-18 22:54:13,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:13,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-18 22:54:14,187 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=742 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/e3694fd4068648f995da68bb47324b9c 2023-04-18 22:54:14,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-18 22:54:14,599 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=742 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/e650cc53726042f393e9f7d56de8b2ee 2023-04-18 22:54:15,010 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=742 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/ec66494ee33e4d4d89c315113189fc6c 2023-04-18 22:54:15,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/A/e3694fd4068648f995da68bb47324b9c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/e3694fd4068648f995da68bb47324b9c 2023-04-18 22:54:15,019 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/e3694fd4068648f995da68bb47324b9c, entries=100, sequenceid=742, filesize=9.5 K 2023-04-18 22:54:15,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/B/e650cc53726042f393e9f7d56de8b2ee as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/e650cc53726042f393e9f7d56de8b2ee 2023-04-18 22:54:15,024 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/e650cc53726042f393e9f7d56de8b2ee, entries=100, sequenceid=742, filesize=9.5 K 2023-04-18 22:54:15,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/.tmp/C/ec66494ee33e4d4d89c315113189fc6c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/ec66494ee33e4d4d89c315113189fc6c 2023-04-18 22:54:15,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/ec66494ee33e4d4d89c315113189fc6c, entries=100, sequenceid=742, filesize=9.5 K 2023-04-18 22:54:15,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 3d5ba4e7a99f0fcc1148990ad0a52e36 in 1252ms, sequenceid=742, compaction requested=true 2023-04-18 22:54:15,057 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/a7b990a2771f4953ae61cd7f43f5ff31, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/7f3eece9566b4492a86d7019498c3e7d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/b9451068b4e34c4fa315bb24d0d1dc5b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0984ce701d884f9cafc20b8d05976fe3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/aa78dcdf9ecf439a97d1d61ed121c6c9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/435d66c40bb5496eb34dfdd6801a8b4a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/38945d35cd61473895b55b26dc59a9f5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/58506a812ab042b695f267284e762257, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/90f0fc2dc8e0488e9bff2abc2b8c5e6e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/4ff19fef60a54d00b139036e8ba78aaf, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c7940a10b5d84ee79ffcc91167afed94, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c9427866bb88435ea4bb55873ffef52e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0e4cbfb051144576861cd409c9293b3d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/361e5015002d480795de7912c022fd2f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/f868a7717baa49a6a647d6168e4a0c83, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/863777c3ad4942f8a9d1fc5ae1eff0a6, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c1470a4367054114b6c0c319ddf12e02, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/35a4684006734133830ea4b440630053, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/36590d18f151415cb7ceb2f821a480c3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/5e4d80031651425c888c14f5be592a2e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/066b7985127a4933b85c0632de1745cd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/578796de2ebb436d93ed97d8b86936f5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/cb51048b934f4396b521d97b4988394e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/e04c1edda8054fa0b28b656ab04234ed, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/e2a275285cf84dce950747f5fc6ee7c7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c9b09c48e28c4c4ead7fe11ceeb89034, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/5d6f3b6b158f4a728e4b6c0c5503be5b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/100d4261877b4cd2a9bc9066ad070687, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/aafdde08168e4d7db03d6310d666db91, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/47d8f132e90046d99ae5aba2bab14d12, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/f71261a8418e46818689a4608f480f97, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/58a68366eed54ed29e5918d7db298337, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/058fd55c1cb245a1aba33af89f92958a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/a75b302f1c984dd4b42d3ef46c9cb795, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0fd4fec072cf43bd9612ad39461b8dc1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/aa3e78baca574ce6a9cf7803247d92c0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c38d189fea0a40a0a1a15182cb34a42c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/4e094655e9c44911af97a215c79e2a5a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/1b2e54a4117a48ea8ea65b842696d750, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0225c5c863684172a9e93990809fa53f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/6511fb58840d4da7a89895faff0e9e6c] to archive 2023-04-18 22:54:15,058 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:54:15,063 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/a7b990a2771f4953ae61cd7f43f5ff31 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/a7b990a2771f4953ae61cd7f43f5ff31 2023-04-18 22:54:15,064 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/7f3eece9566b4492a86d7019498c3e7d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/7f3eece9566b4492a86d7019498c3e7d 2023-04-18 22:54:15,065 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/b9451068b4e34c4fa315bb24d0d1dc5b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/b9451068b4e34c4fa315bb24d0d1dc5b 2023-04-18 22:54:15,067 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0984ce701d884f9cafc20b8d05976fe3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0984ce701d884f9cafc20b8d05976fe3 2023-04-18 22:54:15,068 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/aa78dcdf9ecf439a97d1d61ed121c6c9 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/aa78dcdf9ecf439a97d1d61ed121c6c9 2023-04-18 22:54:15,069 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/435d66c40bb5496eb34dfdd6801a8b4a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/435d66c40bb5496eb34dfdd6801a8b4a 2023-04-18 22:54:15,070 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/38945d35cd61473895b55b26dc59a9f5 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/38945d35cd61473895b55b26dc59a9f5 2023-04-18 22:54:15,071 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/58506a812ab042b695f267284e762257 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/58506a812ab042b695f267284e762257 2023-04-18 22:54:15,073 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/90f0fc2dc8e0488e9bff2abc2b8c5e6e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/90f0fc2dc8e0488e9bff2abc2b8c5e6e 2023-04-18 22:54:15,074 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/4ff19fef60a54d00b139036e8ba78aaf to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/4ff19fef60a54d00b139036e8ba78aaf 2023-04-18 22:54:15,075 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c7940a10b5d84ee79ffcc91167afed94 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c7940a10b5d84ee79ffcc91167afed94 2023-04-18 22:54:15,076 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c9427866bb88435ea4bb55873ffef52e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c9427866bb88435ea4bb55873ffef52e 2023-04-18 22:54:15,077 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0e4cbfb051144576861cd409c9293b3d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0e4cbfb051144576861cd409c9293b3d 2023-04-18 22:54:15,078 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/361e5015002d480795de7912c022fd2f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/361e5015002d480795de7912c022fd2f 2023-04-18 22:54:15,079 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/f868a7717baa49a6a647d6168e4a0c83 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/f868a7717baa49a6a647d6168e4a0c83 2023-04-18 22:54:15,081 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/863777c3ad4942f8a9d1fc5ae1eff0a6 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/863777c3ad4942f8a9d1fc5ae1eff0a6 2023-04-18 22:54:15,082 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c1470a4367054114b6c0c319ddf12e02 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c1470a4367054114b6c0c319ddf12e02 2023-04-18 22:54:15,083 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/35a4684006734133830ea4b440630053 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/35a4684006734133830ea4b440630053 2023-04-18 22:54:15,084 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/36590d18f151415cb7ceb2f821a480c3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/36590d18f151415cb7ceb2f821a480c3 2023-04-18 22:54:15,085 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/5e4d80031651425c888c14f5be592a2e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/5e4d80031651425c888c14f5be592a2e 2023-04-18 22:54:15,086 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/066b7985127a4933b85c0632de1745cd to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/066b7985127a4933b85c0632de1745cd 2023-04-18 22:54:15,087 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/578796de2ebb436d93ed97d8b86936f5 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/578796de2ebb436d93ed97d8b86936f5 2023-04-18 22:54:15,088 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/cb51048b934f4396b521d97b4988394e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/cb51048b934f4396b521d97b4988394e 2023-04-18 22:54:15,089 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/e04c1edda8054fa0b28b656ab04234ed to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/e04c1edda8054fa0b28b656ab04234ed 2023-04-18 22:54:15,090 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/e2a275285cf84dce950747f5fc6ee7c7 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/e2a275285cf84dce950747f5fc6ee7c7 2023-04-18 22:54:15,092 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c9b09c48e28c4c4ead7fe11ceeb89034 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c9b09c48e28c4c4ead7fe11ceeb89034 2023-04-18 22:54:15,093 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/5d6f3b6b158f4a728e4b6c0c5503be5b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/5d6f3b6b158f4a728e4b6c0c5503be5b 2023-04-18 22:54:15,094 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/100d4261877b4cd2a9bc9066ad070687 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/100d4261877b4cd2a9bc9066ad070687 2023-04-18 22:54:15,095 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/aafdde08168e4d7db03d6310d666db91 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/aafdde08168e4d7db03d6310d666db91 2023-04-18 22:54:15,097 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/47d8f132e90046d99ae5aba2bab14d12 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/47d8f132e90046d99ae5aba2bab14d12 2023-04-18 22:54:15,098 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/f71261a8418e46818689a4608f480f97 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/f71261a8418e46818689a4608f480f97 2023-04-18 22:54:15,099 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/58a68366eed54ed29e5918d7db298337 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/58a68366eed54ed29e5918d7db298337 2023-04-18 22:54:15,100 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/058fd55c1cb245a1aba33af89f92958a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/058fd55c1cb245a1aba33af89f92958a 2023-04-18 22:54:15,100 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/a75b302f1c984dd4b42d3ef46c9cb795 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/a75b302f1c984dd4b42d3ef46c9cb795 2023-04-18 22:54:15,101 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0fd4fec072cf43bd9612ad39461b8dc1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0fd4fec072cf43bd9612ad39461b8dc1 2023-04-18 22:54:15,102 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/aa3e78baca574ce6a9cf7803247d92c0 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/aa3e78baca574ce6a9cf7803247d92c0 2023-04-18 22:54:15,104 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c38d189fea0a40a0a1a15182cb34a42c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/c38d189fea0a40a0a1a15182cb34a42c 2023-04-18 22:54:15,105 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/4e094655e9c44911af97a215c79e2a5a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/4e094655e9c44911af97a215c79e2a5a 2023-04-18 22:54:15,105 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/1b2e54a4117a48ea8ea65b842696d750 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/1b2e54a4117a48ea8ea65b842696d750 2023-04-18 22:54:15,107 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0225c5c863684172a9e93990809fa53f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/0225c5c863684172a9e93990809fa53f 2023-04-18 22:54:15,108 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/6511fb58840d4da7a89895faff0e9e6c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/6511fb58840d4da7a89895faff0e9e6c 2023-04-18 22:54:15,132 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/768745411809421f9ccadbe76b1b29d6, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/6e7ba278388d41f28e1eb92b608bb0d7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/924701d5366d4f03a8005978357e0c57, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b5d1369e93854d0889675ff2748275b5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/e4b7ba5b0de4489bb6fd02f701051f77, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/df6b205a4c6346f7bba1ca9906fcce32, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8265452985964572905a8c0112e9a36c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/0aeef7df8f414762b0c4c53ca8a424ae, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b582ac993d4a493280fda7f54a3c1431, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8679216824a0471786cca255f5c5f6dd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/086d26839d374229b8c0b82385944cd6, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8c8b58e6a4ee413b91783e38a9171a43, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b752795d27ed48659d358df9450cf559, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/91f57482fdcf4c9e8eeffc9f826885e8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d52a4e1d300943389a67e2d5e4ce64eb, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/49ea09cf763a4964bf8b8c6bc9000541, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/de7167cfe6a04bd68f2fc2651191b4ff, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d928f41816fd4653a1cdf547137b25ef, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/998446da163845f48d7544da04e64d22, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/becd2133aac14be593c1aa02a9fca1d9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/5ac64f7b2fd1482497cc79fa31c90227, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b6fe8211ad87473a8f017f2eef43e5a8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/f25263e0c06e4b73bf4e310022bdc8e5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/cc4a9b007965466087509b2bf894965b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/c48dd8cd5a1a4ca994232c098f362ba8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/c4d26b4953f64f0186d7c20a12e55f88, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/7e8b9ae792c147ed91efdca6ce495dfa, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/9e4b36e4972c4cf1845dae3f2453717a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/22a5f588b27a4b198cec4098d5ff1280, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/127ab31e0e7e4054bd75d197a3d12d15, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/3084d00cfd7c49d296b85ac4aff0b7ef, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/120ebb6cd7614e169e3048432bcca4cc, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/db20b9ba4020476b9a112a4af96983db, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/93ea721dfedc4e9eaddb81b74037d101, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/37fc8e6877d043b291c4ebad632c0150, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d356e1d83efd4f11ba1f577c14f9fdf6, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/f300db3469f14ceb97b5b34be993a477, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/a5034a66acf144d1b472986fe6cca190, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/7a22b83b8e084af1b07bfb255f855f53, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/c2958adb0e074baa8e7dc4bbbb63a138, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/85dfd051a8614380ba1feb7945b0f34a] to archive 2023-04-18 22:54:15,133 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:54:15,134 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/768745411809421f9ccadbe76b1b29d6 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/768745411809421f9ccadbe76b1b29d6 2023-04-18 22:54:15,135 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/6e7ba278388d41f28e1eb92b608bb0d7 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/6e7ba278388d41f28e1eb92b608bb0d7 2023-04-18 22:54:15,136 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/924701d5366d4f03a8005978357e0c57 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/924701d5366d4f03a8005978357e0c57 2023-04-18 22:54:15,137 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b5d1369e93854d0889675ff2748275b5 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b5d1369e93854d0889675ff2748275b5 2023-04-18 22:54:15,138 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/e4b7ba5b0de4489bb6fd02f701051f77 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/e4b7ba5b0de4489bb6fd02f701051f77 2023-04-18 22:54:15,140 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/df6b205a4c6346f7bba1ca9906fcce32 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/df6b205a4c6346f7bba1ca9906fcce32 2023-04-18 22:54:15,141 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8265452985964572905a8c0112e9a36c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8265452985964572905a8c0112e9a36c 2023-04-18 22:54:15,142 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/0aeef7df8f414762b0c4c53ca8a424ae to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/0aeef7df8f414762b0c4c53ca8a424ae 2023-04-18 22:54:15,143 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b582ac993d4a493280fda7f54a3c1431 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b582ac993d4a493280fda7f54a3c1431 2023-04-18 22:54:15,144 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8679216824a0471786cca255f5c5f6dd to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8679216824a0471786cca255f5c5f6dd 2023-04-18 22:54:15,145 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/086d26839d374229b8c0b82385944cd6 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/086d26839d374229b8c0b82385944cd6 2023-04-18 22:54:15,145 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8c8b58e6a4ee413b91783e38a9171a43 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8c8b58e6a4ee413b91783e38a9171a43 2023-04-18 22:54:15,147 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b752795d27ed48659d358df9450cf559 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b752795d27ed48659d358df9450cf559 2023-04-18 22:54:15,148 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/91f57482fdcf4c9e8eeffc9f826885e8 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/91f57482fdcf4c9e8eeffc9f826885e8 2023-04-18 22:54:15,149 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d52a4e1d300943389a67e2d5e4ce64eb to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d52a4e1d300943389a67e2d5e4ce64eb 2023-04-18 22:54:15,150 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/49ea09cf763a4964bf8b8c6bc9000541 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/49ea09cf763a4964bf8b8c6bc9000541 2023-04-18 22:54:15,151 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/de7167cfe6a04bd68f2fc2651191b4ff to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/de7167cfe6a04bd68f2fc2651191b4ff 2023-04-18 22:54:15,152 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d928f41816fd4653a1cdf547137b25ef to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d928f41816fd4653a1cdf547137b25ef 2023-04-18 22:54:15,154 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/998446da163845f48d7544da04e64d22 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/998446da163845f48d7544da04e64d22 2023-04-18 22:54:15,155 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/becd2133aac14be593c1aa02a9fca1d9 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/becd2133aac14be593c1aa02a9fca1d9 2023-04-18 22:54:15,156 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/5ac64f7b2fd1482497cc79fa31c90227 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/5ac64f7b2fd1482497cc79fa31c90227 2023-04-18 22:54:15,157 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b6fe8211ad87473a8f017f2eef43e5a8 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/b6fe8211ad87473a8f017f2eef43e5a8 2023-04-18 22:54:15,158 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/f25263e0c06e4b73bf4e310022bdc8e5 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/f25263e0c06e4b73bf4e310022bdc8e5 2023-04-18 22:54:15,159 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/cc4a9b007965466087509b2bf894965b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/cc4a9b007965466087509b2bf894965b 2023-04-18 22:54:15,160 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/c48dd8cd5a1a4ca994232c098f362ba8 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/c48dd8cd5a1a4ca994232c098f362ba8 2023-04-18 22:54:15,161 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/c4d26b4953f64f0186d7c20a12e55f88 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/c4d26b4953f64f0186d7c20a12e55f88 2023-04-18 22:54:15,162 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/7e8b9ae792c147ed91efdca6ce495dfa to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/7e8b9ae792c147ed91efdca6ce495dfa 2023-04-18 22:54:15,163 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/9e4b36e4972c4cf1845dae3f2453717a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/9e4b36e4972c4cf1845dae3f2453717a 2023-04-18 22:54:15,164 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/22a5f588b27a4b198cec4098d5ff1280 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/22a5f588b27a4b198cec4098d5ff1280 2023-04-18 22:54:15,165 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/127ab31e0e7e4054bd75d197a3d12d15 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/127ab31e0e7e4054bd75d197a3d12d15 2023-04-18 22:54:15,166 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/3084d00cfd7c49d296b85ac4aff0b7ef to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/3084d00cfd7c49d296b85ac4aff0b7ef 2023-04-18 22:54:15,167 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/120ebb6cd7614e169e3048432bcca4cc to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/120ebb6cd7614e169e3048432bcca4cc 2023-04-18 22:54:15,168 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/db20b9ba4020476b9a112a4af96983db to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/db20b9ba4020476b9a112a4af96983db 2023-04-18 22:54:15,169 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/93ea721dfedc4e9eaddb81b74037d101 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/93ea721dfedc4e9eaddb81b74037d101 2023-04-18 22:54:15,170 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/37fc8e6877d043b291c4ebad632c0150 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/37fc8e6877d043b291c4ebad632c0150 2023-04-18 22:54:15,171 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d356e1d83efd4f11ba1f577c14f9fdf6 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/d356e1d83efd4f11ba1f577c14f9fdf6 2023-04-18 22:54:15,172 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/f300db3469f14ceb97b5b34be993a477 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/f300db3469f14ceb97b5b34be993a477 2023-04-18 22:54:15,173 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/a5034a66acf144d1b472986fe6cca190 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/a5034a66acf144d1b472986fe6cca190 2023-04-18 22:54:15,174 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/7a22b83b8e084af1b07bfb255f855f53 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/7a22b83b8e084af1b07bfb255f855f53 2023-04-18 22:54:15,175 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/c2958adb0e074baa8e7dc4bbbb63a138 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/c2958adb0e074baa8e7dc4bbbb63a138 2023-04-18 22:54:15,176 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/85dfd051a8614380ba1feb7945b0f34a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/85dfd051a8614380ba1feb7945b0f34a 2023-04-18 22:54:15,180 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fcc98a05222a4ce6beab00ee0c756103, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/08bb7a7e62c94ad78d1de72644081ebd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/cbde6625cee14421924167fa988b5a62, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/3e390e11a80f4551bf92115d78e87864, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c87dc38019124e97914e753aa5885ae9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/72c212879d28493b98bbeba2452f7b63, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f996372401674774bfee37c55ff3732a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/197e5f72da9e45d3b5d3a848072fc86d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/faf1319830c1433a995430f9ca9e3261, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/828f87cadcfd4236a4efa39422ca50ca, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6c76f07713c14c2abe68e12f67ff21c2, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/e58f3495bbbb4133941a477191498b30, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c03dba2a81604adfbce9276254a1e03f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/7f744e3edfb941ea857dfe66ef4f689b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/959af490ddd143048421f85b8812fbb8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/98283a2c381a433799ef2584e0a24b7a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/56e3b2cd11c54a37ba100c2b25fcfb84, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fd7435c0747340d4bb054cf32e5def09, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/b34c035620ac4a528c91acce1167f5f1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fe86f138bd27449eaa6d7cab5c74e156, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c0f6680889c140b9b7d75ad6536881e0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/8cb024b05ece434d86ec6b24450ea2fe, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6c17f4ad5dfb47e8ab5d1b7f796d8864, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6cbcda9d5b904acbb806d0a8474fc48c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/5e9590c5c0eb45d7ac0aa4e26302dd6c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1fe50a3c92a74148a3168163bf379a1f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6e4a9ee6f561465eb370896b5bd69d8a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f622670534bb4b8abba967c11ead68c1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/79cec455204b44818f75f6a3bf2b6655, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/dfa1622f677e44d8a26cadb7b683a32a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/11f96da1d10640ea86e698c22e5bc234, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f9f458c8e8724bc996efc6a5a050ac1d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/4fe7952f18884c648f6ca43a2ca88149, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1efc59b97077481a97914596573f34d1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f74cf5f78f744b4fb638409fe6716e04, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/9b813d25952046eaad0ac45bc2d966b3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1c41fee9c61e4f2fb9d2e5b2943142ec, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/a338f060130348efb85902a0cce3987a] to archive 2023-04-18 22:54:15,181 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:54:15,182 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fcc98a05222a4ce6beab00ee0c756103 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fcc98a05222a4ce6beab00ee0c756103 2023-04-18 22:54:15,184 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/08bb7a7e62c94ad78d1de72644081ebd to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/08bb7a7e62c94ad78d1de72644081ebd 2023-04-18 22:54:15,185 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/cbde6625cee14421924167fa988b5a62 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/cbde6625cee14421924167fa988b5a62 2023-04-18 22:54:15,187 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/3e390e11a80f4551bf92115d78e87864 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/3e390e11a80f4551bf92115d78e87864 2023-04-18 22:54:15,188 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c87dc38019124e97914e753aa5885ae9 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c87dc38019124e97914e753aa5885ae9 2023-04-18 22:54:15,190 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/72c212879d28493b98bbeba2452f7b63 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/72c212879d28493b98bbeba2452f7b63 2023-04-18 22:54:15,191 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f996372401674774bfee37c55ff3732a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f996372401674774bfee37c55ff3732a 2023-04-18 22:54:15,193 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/197e5f72da9e45d3b5d3a848072fc86d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/197e5f72da9e45d3b5d3a848072fc86d 2023-04-18 22:54:15,194 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/faf1319830c1433a995430f9ca9e3261 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/faf1319830c1433a995430f9ca9e3261 2023-04-18 22:54:15,196 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/828f87cadcfd4236a4efa39422ca50ca to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/828f87cadcfd4236a4efa39422ca50ca 2023-04-18 22:54:15,197 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6c76f07713c14c2abe68e12f67ff21c2 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6c76f07713c14c2abe68e12f67ff21c2 2023-04-18 22:54:15,199 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/e58f3495bbbb4133941a477191498b30 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/e58f3495bbbb4133941a477191498b30 2023-04-18 22:54:15,201 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c03dba2a81604adfbce9276254a1e03f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c03dba2a81604adfbce9276254a1e03f 2023-04-18 22:54:15,203 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/7f744e3edfb941ea857dfe66ef4f689b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/7f744e3edfb941ea857dfe66ef4f689b 2023-04-18 22:54:15,204 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/959af490ddd143048421f85b8812fbb8 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/959af490ddd143048421f85b8812fbb8 2023-04-18 22:54:15,206 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/98283a2c381a433799ef2584e0a24b7a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/98283a2c381a433799ef2584e0a24b7a 2023-04-18 22:54:15,207 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/56e3b2cd11c54a37ba100c2b25fcfb84 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/56e3b2cd11c54a37ba100c2b25fcfb84 2023-04-18 22:54:15,209 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fd7435c0747340d4bb054cf32e5def09 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fd7435c0747340d4bb054cf32e5def09 2023-04-18 22:54:15,210 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/b34c035620ac4a528c91acce1167f5f1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/b34c035620ac4a528c91acce1167f5f1 2023-04-18 22:54:15,211 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fe86f138bd27449eaa6d7cab5c74e156 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/fe86f138bd27449eaa6d7cab5c74e156 2023-04-18 22:54:15,213 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c0f6680889c140b9b7d75ad6536881e0 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/c0f6680889c140b9b7d75ad6536881e0 2023-04-18 22:54:15,214 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/8cb024b05ece434d86ec6b24450ea2fe to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/8cb024b05ece434d86ec6b24450ea2fe 2023-04-18 22:54:15,215 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6c17f4ad5dfb47e8ab5d1b7f796d8864 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6c17f4ad5dfb47e8ab5d1b7f796d8864 2023-04-18 22:54:15,216 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6cbcda9d5b904acbb806d0a8474fc48c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6cbcda9d5b904acbb806d0a8474fc48c 2023-04-18 22:54:15,217 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/5e9590c5c0eb45d7ac0aa4e26302dd6c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/5e9590c5c0eb45d7ac0aa4e26302dd6c 2023-04-18 22:54:15,219 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1fe50a3c92a74148a3168163bf379a1f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1fe50a3c92a74148a3168163bf379a1f 2023-04-18 22:54:15,220 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6e4a9ee6f561465eb370896b5bd69d8a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6e4a9ee6f561465eb370896b5bd69d8a 2023-04-18 22:54:15,221 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f622670534bb4b8abba967c11ead68c1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f622670534bb4b8abba967c11ead68c1 2023-04-18 22:54:15,223 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/79cec455204b44818f75f6a3bf2b6655 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/79cec455204b44818f75f6a3bf2b6655 2023-04-18 22:54:15,224 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/dfa1622f677e44d8a26cadb7b683a32a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/dfa1622f677e44d8a26cadb7b683a32a 2023-04-18 22:54:15,225 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/11f96da1d10640ea86e698c22e5bc234 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/11f96da1d10640ea86e698c22e5bc234 2023-04-18 22:54:15,227 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f9f458c8e8724bc996efc6a5a050ac1d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f9f458c8e8724bc996efc6a5a050ac1d 2023-04-18 22:54:15,228 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/4fe7952f18884c648f6ca43a2ca88149 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/4fe7952f18884c648f6ca43a2ca88149 2023-04-18 22:54:15,229 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1efc59b97077481a97914596573f34d1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1efc59b97077481a97914596573f34d1 2023-04-18 22:54:15,231 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f74cf5f78f744b4fb638409fe6716e04 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/f74cf5f78f744b4fb638409fe6716e04 2023-04-18 22:54:15,232 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/9b813d25952046eaad0ac45bc2d966b3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/9b813d25952046eaad0ac45bc2d966b3 2023-04-18 22:54:15,233 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1c41fee9c61e4f2fb9d2e5b2943142ec to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/1c41fee9c61e4f2fb9d2e5b2943142ec 2023-04-18 22:54:15,235 DEBUG [StoreCloser-TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/a338f060130348efb85902a0cce3987a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/a338f060130348efb85902a0cce3987a 2023-04-18 22:54:15,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/recovered.edits/745.seqid, newMaxSeqId=745, maxSeqId=1 2023-04-18 22:54:15,242 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36. 2023-04-18 22:54:15,242 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 3d5ba4e7a99f0fcc1148990ad0a52e36: 2023-04-18 22:54:15,244 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:15,244 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=3d5ba4e7a99f0fcc1148990ad0a52e36, regionState=CLOSED 2023-04-18 22:54:15,244 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858455244"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681858455244"}]},"ts":"1681858455244"} 2023-04-18 22:54:15,247 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-18 22:54:15,248 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 3d5ba4e7a99f0fcc1148990ad0a52e36, server=jenkins-hbase4.apache.org,35971,1681858424869 in 1.8740 sec 2023-04-18 22:54:15,249 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-18 22:54:15,249 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3d5ba4e7a99f0fcc1148990ad0a52e36, UNASSIGN in 1.8810 sec 2023-04-18 22:54:15,250 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858455250"}]},"ts":"1681858455250"} 2023-04-18 22:54:15,251 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-18 22:54:15,252 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-18 22:54:15,254 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9000 sec 2023-04-18 22:54:15,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-18 22:54:15,468 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-18 22:54:15,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-18 22:54:15,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:54:15,481 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:54:15,482 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:54:15,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-18 22:54:15,487 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:15,489 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/recovered.edits] 2023-04-18 22:54:15,492 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/5fd9162949894412b79fb943279c73b2 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/5fd9162949894412b79fb943279c73b2 2023-04-18 22:54:15,494 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/e3694fd4068648f995da68bb47324b9c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/A/e3694fd4068648f995da68bb47324b9c 2023-04-18 22:54:15,496 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8f676b22859942eda70d98350a0bea3a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/8f676b22859942eda70d98350a0bea3a 2023-04-18 22:54:15,498 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/e650cc53726042f393e9f7d56de8b2ee to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/B/e650cc53726042f393e9f7d56de8b2ee 2023-04-18 22:54:15,500 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/147410aaff8d4bb1a6e5864887b5f36d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/147410aaff8d4bb1a6e5864887b5f36d 2023-04-18 22:54:15,501 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6daadaec54cd4b61bcf8a50f1da9704f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/6daadaec54cd4b61bcf8a50f1da9704f 2023-04-18 22:54:15,502 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/b4c4739facad40b3bd782eca671b6235 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/b4c4739facad40b3bd782eca671b6235 2023-04-18 22:54:15,503 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/ec66494ee33e4d4d89c315113189fc6c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/C/ec66494ee33e4d4d89c315113189fc6c 2023-04-18 22:54:15,506 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/recovered.edits/745.seqid to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36/recovered.edits/745.seqid 2023-04-18 22:54:15,507 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/3d5ba4e7a99f0fcc1148990ad0a52e36 2023-04-18 22:54:15,507 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 22:54:15,510 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:54:15,531 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-18 22:54:15,581 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-18 22:54:15,583 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:54:15,583 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-18 22:54:15,583 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681858455583"}]},"ts":"9223372036854775807"} 2023-04-18 22:54:15,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-18 22:54:15,590 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-18 22:54:15,590 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3d5ba4e7a99f0fcc1148990ad0a52e36, NAME => 'TestAcidGuarantees,,1681858427817.3d5ba4e7a99f0fcc1148990ad0a52e36.', STARTKEY => '', ENDKEY => ''}] 2023-04-18 22:54:15,590 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-18 22:54:15,590 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681858455590"}]},"ts":"9223372036854775807"} 2023-04-18 22:54:15,592 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-18 22:54:15,595 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:54:15,596 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 121 msec 2023-04-18 22:54:15,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-18 22:54:15,786 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-18 22:54:15,811 INFO [Listener at localhost/36651] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=233 (was 205) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x18afb399-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:35971-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1917238546_15 at /127.0.0.1:48836 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1917238546_15 at /127.0.0.1:48862 [Waiting for operation #9] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x18afb399-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x18afb399-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x18afb399-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=570 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=401 (was 366) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=6475 (was 7202) 2023-04-18 22:54:15,823 INFO [Listener at localhost/36651] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=233, OpenFileDescriptor=570, MaxFileDescriptor=60000, SystemLoadAverage=401, ProcessCount=169, AvailableMemoryMB=6475 2023-04-18 22:54:15,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 22:54:15,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 22:54:15,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-18 22:54:15,829 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-18 22:54:15,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-18 22:54:15,830 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-18 22:54:15,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-18 22:54:15,832 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:15,832 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728 empty. 2023-04-18 22:54:15,833 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:15,833 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 22:54:15,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-18 22:54:16,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-18 22:54:16,244 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 22:54:16,248 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 052b0c71a65b8786b7851f4ab1dab728, NAME => 'TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp 2023-04-18 22:54:16,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-18 22:54:16,658 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:54:16,659 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 052b0c71a65b8786b7851f4ab1dab728, disabling compactions & flushes 2023-04-18 22:54:16,659 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:16,659 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:16,659 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. after waiting 0 ms 2023-04-18 22:54:16,659 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:16,659 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:16,659 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:16,661 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-18 22:54:16,662 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858456662"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681858456662"}]},"ts":"1681858456662"} 2023-04-18 22:54:16,664 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-18 22:54:16,664 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-18 22:54:16,665 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858456665"}]},"ts":"1681858456665"} 2023-04-18 22:54:16,666 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-18 22:54:16,669 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=052b0c71a65b8786b7851f4ab1dab728, ASSIGN}] 2023-04-18 22:54:16,671 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=052b0c71a65b8786b7851f4ab1dab728, ASSIGN 2023-04-18 22:54:16,672 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=052b0c71a65b8786b7851f4ab1dab728, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35971,1681858424869; forceNewPlan=false, retain=false 2023-04-18 22:54:16,823 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=052b0c71a65b8786b7851f4ab1dab728, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:16,823 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858456823"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858456823"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858456823"}]},"ts":"1681858456823"} 2023-04-18 22:54:16,825 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:54:16,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-18 22:54:16,981 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:16,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 052b0c71a65b8786b7851f4ab1dab728, NAME => 'TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.', STARTKEY => '', ENDKEY => ''} 2023-04-18 22:54:16,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:16,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:54:16,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:16,982 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:16,983 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:16,984 DEBUG [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A 2023-04-18 22:54:16,985 DEBUG [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A 2023-04-18 22:54:16,985 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:54:16,985 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 052b0c71a65b8786b7851f4ab1dab728 columnFamilyName A 2023-04-18 22:54:16,986 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.HStore(310): Store=052b0c71a65b8786b7851f4ab1dab728/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:54:16,986 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:16,988 DEBUG [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B 2023-04-18 22:54:16,988 DEBUG [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B 2023-04-18 22:54:16,988 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:54:16,989 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 052b0c71a65b8786b7851f4ab1dab728 columnFamilyName B 2023-04-18 22:54:16,989 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.HStore(310): Store=052b0c71a65b8786b7851f4ab1dab728/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:54:16,990 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:16,991 DEBUG [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C 2023-04-18 22:54:16,991 DEBUG [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C 2023-04-18 22:54:16,992 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:54:16,992 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 052b0c71a65b8786b7851f4ab1dab728 columnFamilyName C 2023-04-18 22:54:16,993 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.HStore(310): Store=052b0c71a65b8786b7851f4ab1dab728/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:54:16,993 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:16,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:16,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:16,999 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 22:54:17,003 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:17,006 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 22:54:17,006 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 052b0c71a65b8786b7851f4ab1dab728; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68527830, jitterRate=0.021144241094589233}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 22:54:17,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:17,007 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., pid=18, masterSystemTime=1681858456977 2023-04-18 22:54:17,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:17,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:17,009 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=052b0c71a65b8786b7851f4ab1dab728, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:17,010 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858457009"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681858457009"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681858457009"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681858457009"}]},"ts":"1681858457009"} 2023-04-18 22:54:17,012 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-18 22:54:17,012 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 in 186 msec 2023-04-18 22:54:17,014 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-18 22:54:17,014 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=052b0c71a65b8786b7851f4ab1dab728, ASSIGN in 343 msec 2023-04-18 22:54:17,014 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-18 22:54:17,014 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858457014"}]},"ts":"1681858457014"} 2023-04-18 22:54:17,015 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-18 22:54:17,017 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-18 22:54:17,019 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1910 sec 2023-04-18 22:54:17,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-18 22:54:17,936 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-18 22:54:17,937 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x33c943f0 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:17,943 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b8d0eb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:17,944 DEBUG [Listener at localhost/36651] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:17,948 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58938, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:17,950 DEBUG [Listener at localhost/36651] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-18 22:54:17,952 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38646, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-18 22:54:17,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 22:54:17,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 22:54:17,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-18 22:54:18,380 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-18 22:54:18,381 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 22:54:18,381 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-18 22:54:18,385 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-18 22:54:18,397 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=052b0c71a65b8786b7851f4ab1dab728, REOPEN/MOVE}] 2023-04-18 22:54:18,398 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=052b0c71a65b8786b7851f4ab1dab728, REOPEN/MOVE 2023-04-18 22:54:18,398 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=052b0c71a65b8786b7851f4ab1dab728, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:18,399 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858458398"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858458398"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858458398"}]},"ts":"1681858458398"} 2023-04-18 22:54:18,400 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:54:18,552 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:18,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 052b0c71a65b8786b7851f4ab1dab728, disabling compactions & flushes 2023-04-18 22:54:18,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:18,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:18,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. after waiting 0 ms 2023-04-18 22:54:18,554 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:18,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-18 22:54:18,558 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:18,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:18,559 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 052b0c71a65b8786b7851f4ab1dab728 to self. 2023-04-18 22:54:18,561 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:18,562 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=052b0c71a65b8786b7851f4ab1dab728, regionState=CLOSED 2023-04-18 22:54:18,562 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858458561"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681858458561"}]},"ts":"1681858458561"} 2023-04-18 22:54:18,565 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-18 22:54:18,565 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 in 163 msec 2023-04-18 22:54:18,566 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=052b0c71a65b8786b7851f4ab1dab728, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35971,1681858424869; forceNewPlan=false, retain=true 2023-04-18 22:54:18,716 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=052b0c71a65b8786b7851f4ab1dab728, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:18,717 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858458716"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858458716"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858458716"}]},"ts":"1681858458716"} 2023-04-18 22:54:18,719 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:54:18,875 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:18,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 052b0c71a65b8786b7851f4ab1dab728, NAME => 'TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.', STARTKEY => '', ENDKEY => ''} 2023-04-18 22:54:18,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:18,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:54:18,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:18,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:18,879 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:18,880 DEBUG [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A 2023-04-18 22:54:18,880 DEBUG [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A 2023-04-18 22:54:18,880 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:54:18,883 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 052b0c71a65b8786b7851f4ab1dab728 columnFamilyName A 2023-04-18 22:54:18,885 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.HStore(310): Store=052b0c71a65b8786b7851f4ab1dab728/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:54:18,886 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:18,887 DEBUG [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B 2023-04-18 22:54:18,887 DEBUG [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B 2023-04-18 22:54:18,887 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:54:18,887 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 052b0c71a65b8786b7851f4ab1dab728 columnFamilyName B 2023-04-18 22:54:18,888 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.HStore(310): Store=052b0c71a65b8786b7851f4ab1dab728/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:54:18,888 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:18,889 DEBUG [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C 2023-04-18 22:54:18,889 DEBUG [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C 2023-04-18 22:54:18,889 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:54:18,890 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 052b0c71a65b8786b7851f4ab1dab728 columnFamilyName C 2023-04-18 22:54:18,890 INFO [StoreOpener-052b0c71a65b8786b7851f4ab1dab728-1] regionserver.HStore(310): Store=052b0c71a65b8786b7851f4ab1dab728/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:54:18,891 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:18,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:18,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:18,894 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 22:54:18,895 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:18,896 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 052b0c71a65b8786b7851f4ab1dab728; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64610547, jitterRate=-0.037227824330329895}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 22:54:18,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:18,897 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., pid=23, masterSystemTime=1681858458872 2023-04-18 22:54:18,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:18,898 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:18,899 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=052b0c71a65b8786b7851f4ab1dab728, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:18,899 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858458899"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681858458899"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681858458899"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681858458899"}]},"ts":"1681858458899"} 2023-04-18 22:54:18,902 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-18 22:54:18,902 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 in 181 msec 2023-04-18 22:54:18,904 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-18 22:54:18,904 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=052b0c71a65b8786b7851f4ab1dab728, REOPEN/MOVE in 505 msec 2023-04-18 22:54:18,907 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-18 22:54:18,907 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 521 msec 2023-04-18 22:54:18,909 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 946 msec 2023-04-18 22:54:18,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-18 22:54:18,920 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x5b3934fb to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:18,925 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34135972, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:18,926 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x1aa5eef5 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:18,931 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a71edf0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:18,932 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x1441c9f0 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:18,935 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21265fcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:18,936 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x69f15cb4 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:18,943 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7aba8e46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:18,944 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x6c1c92c6 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:18,948 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@feb770d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:18,948 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x54451f11 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:18,954 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c2598b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:18,955 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x2c4379c2 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:18,959 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@508246cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:18,960 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x3f43b712 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:18,965 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8795451, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:18,966 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x7a3bb940 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:18,972 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58e27a32, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:18,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:18,986 DEBUG [hconnection-0x2811246f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:18,987 DEBUG [hconnection-0x7e42a283-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:18,987 DEBUG [hconnection-0x144d05a7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:18,987 DEBUG [hconnection-0x1e24ccda-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:18,987 DEBUG [hconnection-0x77e75aa1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:18,989 DEBUG [hconnection-0x77d4528d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:18,989 DEBUG [hconnection-0x6fa4f6cd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:18,994 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58968, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:18,994 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58972, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:18,994 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58988, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:18,994 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58994, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:18,994 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58954, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:18,994 DEBUG [hconnection-0x7490563f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:18,994 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58974, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:18,995 DEBUG [hconnection-0x7b6b74a9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:18,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:18,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:18,995 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:18,995 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:18,995 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:18,995 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:18,997 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:18,997 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:18,997 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59006, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:18,997 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59014, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:19,001 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59026, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:19,004 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,004 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:19,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:19,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:19,009 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,009 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:19,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:19,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:19,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:19,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:19,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:19,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:19,023 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:19,023 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:19,023 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:19,025 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:19,025 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:19,026 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:19,026 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:19,026 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-18 22:54:19,027 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:19,027 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:19,027 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:19,027 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:19,027 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:19,027 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:19,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:19,029 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:19,067 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418e7fee6aa86dd413e9977335c1aaf173e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e7fee6aa86dd413e9977335c1aaf173e 2023-04-18 22:54:19,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681858459048, max timestamp=1681858459080Num uniques 859; 2023-04-18 22:54:19,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-18 22:54:19,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681858459048, max timestamp=1681858459080Num uniques 859; 2023-04-18 22:54:19,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-18 22:54:19,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681858459048, max timestamp=1681858459081Num uniques 859; 2023-04-18 22:54:19,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-18 22:54:19,090 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=13, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/03602746bd4d4194b4394f2e6a67c63b 2023-04-18 22:54:19,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858519105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858519106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858519107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858519108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858519110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858519215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858519215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858519215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858519215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858519216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858519419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858519419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858519420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858519420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858519418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,553 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/0d20ac7e35294aec841e8f25615d59ed 2023-04-18 22:54:19,576 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/e251d601ef694fadb8c46576eb9b62cc 2023-04-18 22:54:19,585 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/03602746bd4d4194b4394f2e6a67c63b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/03602746bd4d4194b4394f2e6a67c63b 2023-04-18 22:54:19,591 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/03602746bd4d4194b4394f2e6a67c63b, entries=150, sequenceid=13, filesize=25.1 K 2023-04-18 22:54:19,592 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/0d20ac7e35294aec841e8f25615d59ed as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/0d20ac7e35294aec841e8f25615d59ed 2023-04-18 22:54:19,603 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/0d20ac7e35294aec841e8f25615d59ed, entries=150, sequenceid=13, filesize=11.6 K 2023-04-18 22:54:19,605 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/e251d601ef694fadb8c46576eb9b62cc as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e251d601ef694fadb8c46576eb9b62cc 2023-04-18 22:54:19,618 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e251d601ef694fadb8c46576eb9b62cc, entries=150, sequenceid=13, filesize=11.6 K 2023-04-18 22:54:19,619 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 052b0c71a65b8786b7851f4ab1dab728 in 593ms, sequenceid=13, compaction requested=false 2023-04-18 22:54:19,619 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-18 22:54:19,623 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:19,623 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:19,623 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:19,623 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:19,623 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,626 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:19,626 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:19,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:19,628 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:19,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:19,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:19,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:19,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@413d8aee[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:19,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:19,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:19,632 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:19,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:19,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:19,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:19,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:19,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:19,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:19,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:19,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:19,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:19,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:19,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:19,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:19,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:19,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:19,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:19,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:19,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:19,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:19,637 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:19,637 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:19,637 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:19,637 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:19,637 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:19,641 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:19,641 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:19,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:19,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:19,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:19,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:19,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:19,643 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:19,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:19,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:19,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:19,644 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:19,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:19,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:19,646 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:19,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,650 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:19,650 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:19,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:19,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:19,651 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:19,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:19,651 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:19,652 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:19,652 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,652 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:19,652 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:19,652 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:19,652 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:19,652 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:19,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:19,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:19,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:19,651 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:19,736 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=154.31 KB heapSize=426 KB 2023-04-18 22:54:19,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:19,744 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:19,744 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:19,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:19,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:19,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:19,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:19,752 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:19,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:19,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:19,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858519765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858519766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:19,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:19,800 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:19,800 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:19,800 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:19,800 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:19,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:19,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:19,805 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:19,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:19,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:19,806 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:19,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:19,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:19,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:19,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:19,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:19,813 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:19,814 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:19,814 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:19,814 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:19,814 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:19,814 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:19,814 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:19,814 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:19,814 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,814 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:19,814 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:19,816 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,817 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:19,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:19,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:19,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:19,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3470c63[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:19,820 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:19,820 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:19,823 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:19,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:19,823 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:19,823 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:19,824 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:19,824 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,824 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:19,825 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,825 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:19,825 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:19,825 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:19,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:19,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:19,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:19,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858519781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858519789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:19,830 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:19,830 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:19,830 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:19,830 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:19,830 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:19,831 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:19,831 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:19,832 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:19,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:19,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:19,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:19,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:19,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:19,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:19,832 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,833 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858519796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:19,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:19,835 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:19,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:19,836 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:19,836 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,836 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:19,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:19,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:19,836 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:19,836 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,836 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:19,836 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:19,836 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:19,836 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:19,836 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:19,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:19,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858519898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858519898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,909 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418400e8df518f74b35981a1e014479aad3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418400e8df518f74b35981a1e014479aad3 2023-04-18 22:54:19,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858519930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,936 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:19,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:19,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858519935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:19,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858519936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:19,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:19,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:19,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:19,996 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:19,996 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:19,996 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:19,996 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:19,997 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:19,997 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:19,999 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:19,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:19,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,000 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,000 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:20,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:20,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:20,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:20,020 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:20,020 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:20,020 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:20,028 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:20,028 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:20,028 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:20,028 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:20,028 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:20,028 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:20,032 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:20,033 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:20,033 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:20,049 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,049 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,049 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,049 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:20,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dc0945f[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:20,051 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:20,051 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,053 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,053 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:20,053 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:20,053 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:20,053 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,053 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:20,056 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:20,056 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:20,056 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:20,056 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:20,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:20,060 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:20,060 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:20,060 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:20,060 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:20,060 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:20,062 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,062 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:20,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:20,062 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,063 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,064 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,066 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:20,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:20,066 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:20,066 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:20,066 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,067 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,067 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,067 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,067 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,067 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,066 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:20,067 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,066 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:20,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:20,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:20,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:20,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:20,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858520107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:20,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858520107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:20,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858520135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:20,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:20,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858520146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858520146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,167 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:20,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:20,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:20,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:20,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:20,179 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:20,179 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:20,179 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:20,179 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:20,180 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,180 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,181 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:20,181 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:20,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:20,182 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,182 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:20,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:20,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:20,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:20,188 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:20,188 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:20,188 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:20,188 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:20,188 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:20,188 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:20,188 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:20,188 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:20,188 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,188 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:20,189 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:20,192 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,194 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,200 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,200 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:20,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@342ad7ea[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:20,202 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:20,202 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,203 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:20,204 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,204 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:20,205 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,205 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:20,206 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:20,206 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:20,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:20,210 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:20,210 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:20,210 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:20,210 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:20,210 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:20,212 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,212 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,212 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:20,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,213 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,214 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:20,218 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:20,218 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:20,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:20,218 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:20,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:20,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:20,218 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,219 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,219 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,219 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:20,219 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,219 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,219 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,219 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:20,219 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:20,319 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:20,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:20,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:20,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:20,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:20,324 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:20,324 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:20,324 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:20,324 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:20,324 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,324 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,325 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,325 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:20,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,326 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,326 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:20,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:20,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:20,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:20,327 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:20,327 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:20,327 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:20,327 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:20,327 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:20,327 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:20,328 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:20,328 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:20,328 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:20,329 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,329 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,330 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,330 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:20,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@196d517b[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:20,331 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:20,331 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,332 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,332 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:20,332 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:20,332 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:20,332 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,333 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:20,337 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,337 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:20,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:20,340 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:20,340 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:20,340 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:20,340 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:20,340 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:20,342 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,342 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,342 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:20,342 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,343 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:20,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:20,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:20,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:20,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:20,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,346 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:20,347 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:20,347 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:20,347 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,347 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:20,347 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:20,387 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/483dcda33a47434a9e39b26ac6056208 2023-04-18 22:54:20,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:20,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858520410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:20,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858520414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,447 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:20,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:20,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:20,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858520451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:20,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:20,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858520452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:20,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858520455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:20,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:20,468 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:20,468 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:20,468 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:20,468 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:20,469 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,469 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,470 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,470 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:20,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:20,471 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,471 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:20,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:20,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:20,472 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:20,472 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:20,472 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:20,481 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:20,481 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:20,481 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:20,481 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:20,481 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:20,481 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:20,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:20,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:20,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:20,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:20,485 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:20,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22f27221[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:20,488 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:20,488 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,489 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,489 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:20,490 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,490 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,491 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,491 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:20,491 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:20,491 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:20,491 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:20,496 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:20,496 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:20,496 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:20,502 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:20,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:20,505 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:20,505 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:20,505 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:20,505 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:20,505 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:20,508 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,508 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,508 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,510 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,513 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,515 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:20,516 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,516 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,516 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:20,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:20,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:20,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:20,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:20,516 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:20,516 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,516 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,516 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,516 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:20,516 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,516 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,516 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,516 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:20,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:20,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:20,616 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:20,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:20,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:20,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:20,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:20,622 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:20,622 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:20,622 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:20,623 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:20,623 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,623 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,624 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,624 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:20,629 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:20,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:20,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:20,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:20,631 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:20,631 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:20,631 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:20,631 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:20,631 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:20,631 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:20,631 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:20,632 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:20,632 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:20,633 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,633 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:20,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b29784c[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:20,636 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:20,636 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,638 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:20,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:20,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:20,638 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,638 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:20,640 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,640 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:20,640 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:20,640 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:20,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:20,643 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:20,643 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:20,643 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:20,643 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:20,643 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:20,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:20,645 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,646 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,649 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:20,649 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,649 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,649 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,649 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:20,649 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:20,649 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:20,649 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,649 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,650 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:20,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:20,650 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:20,650 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:20,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:20,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:20,750 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:20,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:20,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:20,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:20,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:20,756 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:20,756 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:20,756 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:20,756 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:20,757 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,757 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,758 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:20,758 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,759 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,759 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:20,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:20,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:20,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:20,760 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:20,760 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:20,760 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:20,760 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:20,760 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:20,760 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:20,761 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:20,761 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:20,761 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:20,761 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:20,761 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:20,761 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,761 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:20,761 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:20,765 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,765 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,766 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,766 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:20,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@795e7944[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:20,768 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:20,768 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,769 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,770 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:20,770 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:20,770 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:20,770 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,770 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:20,776 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:20,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:20,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:20,779 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:20,779 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:20,779 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:20,779 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:20,779 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:20,782 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,782 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,782 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:20,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:20,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:20,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:20,784 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,786 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,788 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:20,788 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:20,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:20,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:20,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:20,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:20,788 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,788 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,788 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,788 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,789 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:20,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:20,789 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:20,789 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:20,788 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:20,789 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:20,789 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,801 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-18 22:54:20,808 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/e43b786959154461bbdb861f21e3858f 2023-04-18 22:54:20,855 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/d82bf489bd1c44dd8b97d3399f346503 2023-04-18 22:54:20,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/483dcda33a47434a9e39b26ac6056208 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/483dcda33a47434a9e39b26ac6056208 2023-04-18 22:54:20,868 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/483dcda33a47434a9e39b26ac6056208, entries=150, sequenceid=50, filesize=25.1 K 2023-04-18 22:54:20,869 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/e43b786959154461bbdb861f21e3858f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/e43b786959154461bbdb861f21e3858f 2023-04-18 22:54:20,878 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/e43b786959154461bbdb861f21e3858f, entries=150, sequenceid=50, filesize=11.6 K 2023-04-18 22:54:20,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/d82bf489bd1c44dd8b97d3399f346503 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/d82bf489bd1c44dd8b97d3399f346503 2023-04-18 22:54:20,889 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:20,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:20,891 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/d82bf489bd1c44dd8b97d3399f346503, entries=150, sequenceid=50, filesize=11.6 K 2023-04-18 22:54:20,892 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~445.88 KB/456576, currentSize=33.54 KB/34350 for 052b0c71a65b8786b7851f4ab1dab728 in 1156ms, sequenceid=50, compaction requested=false 2023-04-18 22:54:20,893 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:20,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:20,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:20,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:20,897 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:20,897 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:20,897 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:20,898 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:20,898 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,898 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,899 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,899 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:20,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:20,900 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:20,900 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:20,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:20,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:20,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:20,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:20,904 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:20,904 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:20,904 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:20,905 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:20,905 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:20,905 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:20,905 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:20,905 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-18 22:54:20,905 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:20,906 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:20,906 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:20,906 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:20,906 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:20,906 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:20,952 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041815223920933045df9e4cf0e550ce91f4 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041815223920933045df9e4cf0e550ce91f4 2023-04-18 22:54:20,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:20,961 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:20,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681858460977, max timestamp=1681858460988Num uniques 859; 2023-04-18 22:54:20,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-18 22:54:20,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681858460977, max timestamp=1681858460988Num uniques 859; 2023-04-18 22:54:20,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681858460977, max timestamp=1681858460988Num uniques 859; 2023-04-18 22:54:20,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-18 22:54:20,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-18 22:54:21,006 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=58, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/12bdea8a36b149f398169007c86c7cc0 2023-04-18 22:54:21,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858521007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858521008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858521024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858521024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858521026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,032 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/e326546dc05744f09be9c7f0c7742466 2023-04-18 22:54:21,055 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/65ced05abf394198bb4d92512d264758 2023-04-18 22:54:21,061 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/12bdea8a36b149f398169007c86c7cc0 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/12bdea8a36b149f398169007c86c7cc0 2023-04-18 22:54:21,067 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/12bdea8a36b149f398169007c86c7cc0, entries=100, sequenceid=58, filesize=18.4 K 2023-04-18 22:54:21,068 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/e326546dc05744f09be9c7f0c7742466 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/e326546dc05744f09be9c7f0c7742466 2023-04-18 22:54:21,074 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/e326546dc05744f09be9c7f0c7742466, entries=100, sequenceid=58, filesize=9.3 K 2023-04-18 22:54:21,075 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/65ced05abf394198bb4d92512d264758 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/65ced05abf394198bb4d92512d264758 2023-04-18 22:54:21,084 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/65ced05abf394198bb4d92512d264758, entries=100, sequenceid=58, filesize=9.3 K 2023-04-18 22:54:21,086 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=161.02 KB/164880 for 052b0c71a65b8786b7851f4ab1dab728 in 181ms, sequenceid=58, compaction requested=true 2023-04-18 22:54:21,086 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:21,086 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:21,086 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:21,086 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:21,086 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,086 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:21,086 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,088 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,088 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:21,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a4b199d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:21,092 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:21,092 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,093 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,093 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:21,093 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:21,093 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:21,093 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,094 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:21,095 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:21,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:21,100 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:21,100 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:21,100 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:21,100 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:21,100 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:21,106 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,106 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:21,106 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,107 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,108 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,111 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,111 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,111 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,111 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,111 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,111 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:21,111 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:21,111 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:21,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:21,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:21,111 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:21,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:21,111 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,111 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:21,111 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:21,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:21,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:21,131 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=174.43 KB heapSize=481.78 KB 2023-04-18 22:54:21,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681858461000, max timestamp=1681858461130Num uniques 1268; 2023-04-18 22:54:21,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681858461000, max timestamp=1681858461130Num uniques 1268; 2023-04-18 22:54:21,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=58068, new segment data size=9978, suffix heap size=159840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1268, new segment cells count=218 2023-04-18 22:54:21,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681858461000, max timestamp=1681858461130Num uniques 1268; 2023-04-18 22:54:21,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=58068, new segment data size=9978, suffix heap size=159840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1268, new segment cells count=218 2023-04-18 22:54:21,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=58068, new segment data size=9978, suffix heap size=159840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1268, new segment cells count=218 2023-04-18 22:54:21,137 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:21,137 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:21,137 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:21,137 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:21,137 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:21,137 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:21,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858461151, max timestamp=1681858461161Num uniques 859; 2023-04-18 22:54:21,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:21,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858461151, max timestamp=1681858461161Num uniques 859; 2023-04-18 22:54:21,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:21,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858461151, max timestamp=1681858461161Num uniques 859; 2023-04-18 22:54:21,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:21,171 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041848094233ca3b4cf2a80941d91fa5ce55 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041848094233ca3b4cf2a80941d91fa5ce55 2023-04-18 22:54:21,183 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/e84a91fed94046f3a83cdcc598ba04df 2023-04-18 22:54:21,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858521184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858521187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681858461181, max timestamp=1681858461190Num uniques 1068; 2023-04-18 22:54:21,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-18 22:54:21,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681858461181, max timestamp=1681858461190Num uniques 1068; 2023-04-18 22:54:21,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-18 22:54:21,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681858461181, max timestamp=1681858461190Num uniques 1068; 2023-04-18 22:54:21,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-18 22:54:21,201 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/b323964518d94a2e92d4c5ee921a1d8b 2023-04-18 22:54:21,212 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:21,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:21,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:21,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858521217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858521218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:21,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:21,224 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:21,224 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:21,224 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:21,224 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:21,225 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,225 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,226 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:21,226 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,227 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,227 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:21,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858521223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:21,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:21,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:21,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:21,234 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:21,234 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:21,234 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:21,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:21,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:21,240 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:21,240 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:21,240 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:21,240 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:21,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:21,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:21,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:21,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:21,242 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,242 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,243 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,243 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:21,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b99118f[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:21,245 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:21,245 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,247 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,248 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:21,248 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:21,248 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:21,248 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,248 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:21,249 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,249 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:21,249 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,249 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:21,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:21,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:21,252 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:21,252 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:21,252 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:21,252 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:21,252 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:21,254 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,254 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:21,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,254 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,255 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,256 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,258 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,258 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,258 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,258 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,259 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,259 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:21,258 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:21,259 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:21,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:21,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:21,259 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:21,259 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:21,259 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:21,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:21,259 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:21,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858521292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858521293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858521324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858521324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858521329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,359 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:21,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:21,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:21,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:21,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:21,368 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:21,368 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:21,368 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:21,368 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:21,369 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,369 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,370 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:21,370 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:21,370 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,370 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:21,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:21,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:21,372 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:21,372 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:21,372 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:21,372 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:21,373 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:21,374 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,374 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,374 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,374 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:21,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ba2e5a8[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:21,376 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:21,376 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,377 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,377 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:21,377 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:21,377 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:21,377 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,377 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:21,379 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:21,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:21,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:21,382 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:21,382 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:21,382 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:21,382 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:21,382 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:21,383 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,384 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,384 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:21,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,384 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,386 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,388 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,388 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,388 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:21,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:21,388 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:21,388 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:21,388 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:21,388 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,388 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:21,388 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,388 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:21,388 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:21,388 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:21,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:21,488 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:21,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:21,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:21,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:21,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:21,494 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:21,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:21,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:21,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:21,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,496 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,496 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:21,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:21,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:21,497 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:21,497 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:21,497 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:21,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858521496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858521497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,499 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:21,499 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:21,499 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:21,499 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:21,499 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:21,499 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:21,499 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:21,499 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:21,499 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,499 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:21,499 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:21,504 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,504 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:21,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61bc35c1[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:21,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:21,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,508 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,508 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:21,508 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:21,508 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:21,508 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,508 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:21,510 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:21,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:21,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:21,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:21,515 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:21,515 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:21,515 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:21,515 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:21,515 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:21,516 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,516 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,517 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:21,517 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:21,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:21,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:21,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:21,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:21,525 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:21,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,526 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:21,526 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:21,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:21,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858521530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858521531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858521531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,626 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:21,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:21,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:21,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:21,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:21,631 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:21,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:21,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:21,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:21,632 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,632 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,633 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:21,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:21,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:21,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:21,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:21,635 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:21,635 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:21,635 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:21,635 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:21,635 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:21,635 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,636 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:21,636 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:21,637 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/46bc37e2e335411d96b6a28eb47b284f 2023-04-18 22:54:21,637 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,637 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:21,638 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:21,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33e94f9a[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:21,640 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:21,640 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,643 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/e84a91fed94046f3a83cdcc598ba04df as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/e84a91fed94046f3a83cdcc598ba04df 2023-04-18 22:54:21,645 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,646 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:21,646 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:21,646 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:21,647 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,647 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:21,649 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:21,649 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,649 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,649 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:21,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:21,652 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:21,652 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:21,652 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:21,652 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:21,652 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:21,654 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,654 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,654 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:21,655 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/e84a91fed94046f3a83cdcc598ba04df, entries=150, sequenceid=98, filesize=25.1 K 2023-04-18 22:54:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,657 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/b323964518d94a2e92d4c5ee921a1d8b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b323964518d94a2e92d4c5ee921a1d8b 2023-04-18 22:54:21,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:21,657 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,660 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:21,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:21,663 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:21,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:21,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:21,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:21,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,663 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:21,663 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:21,663 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b323964518d94a2e92d4c5ee921a1d8b, entries=150, sequenceid=98, filesize=11.6 K 2023-04-18 22:54:21,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:21,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,665 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/46bc37e2e335411d96b6a28eb47b284f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/46bc37e2e335411d96b6a28eb47b284f 2023-04-18 22:54:21,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:21,674 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/46bc37e2e335411d96b6a28eb47b284f, entries=150, sequenceid=98, filesize=11.6 K 2023-04-18 22:54:21,675 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~120.42 KB/123312, currentSize=140.89 KB/144270 for 052b0c71a65b8786b7851f4ab1dab728 in 544ms, sequenceid=98, compaction requested=true 2023-04-18 22:54:21,675 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:21,675 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:21,675 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:21,675 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:54:21,675 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:54:21,675 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:21,678 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:54:21,678 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 96036 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:54:21,678 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/B is initiating minor compaction (all files) 2023-04-18 22:54:21,679 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/A is initiating minor compaction (all files) 2023-04-18 22:54:21,679 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/B in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:21,679 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/A in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:21,679 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/0d20ac7e35294aec841e8f25615d59ed, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/e43b786959154461bbdb861f21e3858f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/e326546dc05744f09be9c7f0c7742466, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b323964518d94a2e92d4c5ee921a1d8b] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=44.1 K 2023-04-18 22:54:21,679 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/03602746bd4d4194b4394f2e6a67c63b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/483dcda33a47434a9e39b26ac6056208, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/12bdea8a36b149f398169007c86c7cc0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/e84a91fed94046f3a83cdcc598ba04df] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=93.8 K 2023-04-18 22:54:21,679 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 0d20ac7e35294aec841e8f25615d59ed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681858459012 2023-04-18 22:54:21,679 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 03602746bd4d4194b4394f2e6a67c63b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681858459012 2023-04-18 22:54:21,680 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting e43b786959154461bbdb861f21e3858f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681858459106 2023-04-18 22:54:21,680 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 483dcda33a47434a9e39b26ac6056208, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681858459106 2023-04-18 22:54:21,680 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting e326546dc05744f09be9c7f0c7742466, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681858459766 2023-04-18 22:54:21,680 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 12bdea8a36b149f398169007c86c7cc0, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681858459766 2023-04-18 22:54:21,680 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting b323964518d94a2e92d4c5ee921a1d8b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681858461130 2023-04-18 22:54:21,682 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting e84a91fed94046f3a83cdcc598ba04df, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681858461130 2023-04-18 22:54:21,691 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#B#compaction#140 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:21,718 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#A#compaction#141 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:21,727 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/01a57e5e38544c68bd248124df3c5e2c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/01a57e5e38544c68bd248124df3c5e2c 2023-04-18 22:54:21,733 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/B of 052b0c71a65b8786b7851f4ab1dab728 into 01a57e5e38544c68bd248124df3c5e2c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:21,733 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:21,733 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/B, priority=12, startTime=1681858461675; duration=0sec 2023-04-18 22:54:21,733 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:21,733 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:54:21,735 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:54:21,735 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/C is initiating minor compaction (all files) 2023-04-18 22:54:21,735 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/C in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:21,735 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e251d601ef694fadb8c46576eb9b62cc, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/d82bf489bd1c44dd8b97d3399f346503, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/65ced05abf394198bb4d92512d264758, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/46bc37e2e335411d96b6a28eb47b284f] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=44.1 K 2023-04-18 22:54:21,735 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting e251d601ef694fadb8c46576eb9b62cc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681858459012 2023-04-18 22:54:21,740 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting d82bf489bd1c44dd8b97d3399f346503, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681858459106 2023-04-18 22:54:21,740 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 65ced05abf394198bb4d92512d264758, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681858459766 2023-04-18 22:54:21,741 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 46bc37e2e335411d96b6a28eb47b284f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681858461130 2023-04-18 22:54:21,763 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:21,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:21,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:21,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:21,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:21,771 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:21,771 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:21,771 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:21,771 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:21,772 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,772 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,772 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#C#compaction#142 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:21,773 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:21,773 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:21,774 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,774 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:21,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:21,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:21,775 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:21,775 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:21,775 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:21,776 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:21,776 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:21,776 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=140.89 KB heapSize=394.69 KB 2023-04-18 22:54:21,776 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:21,777 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:21,777 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:21,777 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:21,777 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:21,777 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:21,792 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/828e6b67ad614ec7b6c7e34e7213e1ec as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/828e6b67ad614ec7b6c7e34e7213e1ec 2023-04-18 22:54:21,798 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/060aa7cdbdc94d428e4c284d7bb9ff28 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/060aa7cdbdc94d428e4c284d7bb9ff28 2023-04-18 22:54:21,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:21,805 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:21,818 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/A of 052b0c71a65b8786b7851f4ab1dab728 into 060aa7cdbdc94d428e4c284d7bb9ff28(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:21,818 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:21,818 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/A, priority=12, startTime=1681858461675; duration=0sec 2023-04-18 22:54:21,818 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:21,819 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/C of 052b0c71a65b8786b7851f4ab1dab728 into 828e6b67ad614ec7b6c7e34e7213e1ec(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:21,819 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:21,819 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/C, priority=12, startTime=1681858461675; duration=0sec 2023-04-18 22:54:21,819 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:21,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858521816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858521819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,829 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041839bd796790ff47cf8e0e02970210ead1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041839bd796790ff47cf8e0e02970210ead1 2023-04-18 22:54:21,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858521834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858521835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:21,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858521835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,847 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=152, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/a06d5ad76ac54652be3830b9eb108fbb 2023-04-18 22:54:21,868 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/b5cd34dc3c3c41bcb5be3cbc442feddd 2023-04-18 22:54:21,886 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/bb1c64134f12450dad52c348b90e06f2 2023-04-18 22:54:21,891 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/a06d5ad76ac54652be3830b9eb108fbb as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/a06d5ad76ac54652be3830b9eb108fbb 2023-04-18 22:54:21,897 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/a06d5ad76ac54652be3830b9eb108fbb, entries=150, sequenceid=152, filesize=25.3 K 2023-04-18 22:54:21,898 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/b5cd34dc3c3c41bcb5be3cbc442feddd as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b5cd34dc3c3c41bcb5be3cbc442feddd 2023-04-18 22:54:21,902 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b5cd34dc3c3c41bcb5be3cbc442feddd, entries=150, sequenceid=152, filesize=11.8 K 2023-04-18 22:54:21,903 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/bb1c64134f12450dad52c348b90e06f2 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bb1c64134f12450dad52c348b90e06f2 2023-04-18 22:54:21,907 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bb1c64134f12450dad52c348b90e06f2, entries=150, sequenceid=152, filesize=11.8 K 2023-04-18 22:54:21,909 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~394.64 KB/404112, currentSize=40.25 KB/41220 for 052b0c71a65b8786b7851f4ab1dab728 in 133ms, sequenceid=152, compaction requested=false 2023-04-18 22:54:21,909 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:21,909 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:21,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:21,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:21,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:21,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:21,911 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,911 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,912 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,912 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:21,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@400f2501[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:21,914 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:21,914 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,916 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:21,917 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,917 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:21,918 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:21,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:21,924 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:21,924 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:21,924 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:21,924 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:21,924 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:21,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:21,925 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:54:21,926 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:21,927 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:21,927 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:21,927 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:21,927 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:21,927 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:21,927 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,927 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,927 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:21,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:21,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,929 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,931 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,985 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:21,985 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:21,985 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:21,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:21,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:21,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:21,986 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:21,986 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:21,985 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,986 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:21,986 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:21,986 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,986 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:21,986 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:21,985 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:21,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:22,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858461998, max timestamp=1681858462008Num uniques 859; 2023-04-18 22:54:22,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858461998, max timestamp=1681858462008Num uniques 859; 2023-04-18 22:54:22,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:22,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:22,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858461998, max timestamp=1681858462008Num uniques 859; 2023-04-18 22:54:22,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:22,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:22,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858522046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:22,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858522047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,086 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:22,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:22,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:22,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:22,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:22,089 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:22,089 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:22,089 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:22,089 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:22,090 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,090 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,091 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,091 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:22,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:22,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:22,091 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,091 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:22,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:22,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:22,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:22,092 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:22,092 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:22,093 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:22,093 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:22,093 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:22,093 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:22,093 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,093 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:22,093 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:22,094 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,094 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:22,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:22,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:22,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:22,096 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:22,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c7c070d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:22,096 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,098 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,099 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:22,099 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:22,099 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:22,099 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,099 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:22,100 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:22,100 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,100 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:22,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:22,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:22,100 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:22,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:22,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:22,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:22,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:22,103 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:22,103 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:22,103 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:22,103 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:22,103 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:22,106 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,106 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:22,106 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:22,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:22,107 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,109 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:22,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:22,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:22,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:22,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:22,122 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:22,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:22,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:22,123 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:22,123 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:22,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:22,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858522155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:22,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858522155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,223 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:22,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:22,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:22,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:22,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:22,226 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:22,226 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:22,226 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:22,226 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:22,227 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,227 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,228 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:22,228 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:22,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:22,228 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,229 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:22,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:22,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:22,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:22,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:22,230 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:22,230 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:22,230 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:22,230 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:22,230 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:22,230 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:22,230 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:22,230 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:22,230 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:22,230 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:22,230 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:22,230 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,230 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:22,230 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:22,232 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,232 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:22,232 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,232 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:22,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:22,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:22,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c9c207a[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:22,233 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:22,234 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,235 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,236 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:22,236 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:22,236 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:22,236 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,236 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:22,237 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,237 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:22,237 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:22,237 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:22,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:22,240 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:22,240 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:22,240 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:22,240 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:22,240 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:22,241 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:22,241 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,241 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:22,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:22,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:22,242 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,243 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,247 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,247 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,247 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,247 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,247 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,247 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,247 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:22,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:22,247 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:22,247 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:22,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:22,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:22,247 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:22,248 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:22,248 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:22,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:22,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:22,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858522340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:22,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858522343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:22,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858522343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,348 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:22,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:22,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:22,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:22,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:22,352 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:22,352 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:22,352 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:22,352 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:22,352 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,352 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,354 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,354 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:22,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:22,355 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,355 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:22,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:22,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:22,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:22,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:22,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:22,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:22,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:22,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:22,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:22,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:22,357 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:22,357 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:22,357 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:22,357 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:22,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:22,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:22,358 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,358 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:22,358 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:22,360 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,360 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:22,360 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:22,360 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:22,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:22,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858522360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:22,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:22,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858522360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2084148b[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:22,362 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:22,363 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,364 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,364 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:22,364 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:22,364 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:22,364 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,364 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:22,365 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,365 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:22,366 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:22,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:22,366 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:22,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:22,369 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:22,369 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:22,369 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:22,369 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:22,369 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:22,372 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,372 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,372 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:22,373 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,376 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,378 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:22,378 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:22,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:22,378 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:22,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:22,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:22,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:22,379 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:22,379 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:22,380 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,380 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,380 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,380 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,380 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,380 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:22,380 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,398 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418f1326efe248546efb089cb4336868c8c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f1326efe248546efb089cb4336868c8c 2023-04-18 22:54:22,405 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=165, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/b05097add6e849a584b8db900ca1aa4e 2023-04-18 22:54:22,431 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/c362bbfc5ac64e7b93ae5f5ecb9bc380 2023-04-18 22:54:22,454 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/c993cf7cd86342d1b51bee89ea820a4a 2023-04-18 22:54:22,460 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/b05097add6e849a584b8db900ca1aa4e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/b05097add6e849a584b8db900ca1aa4e 2023-04-18 22:54:22,467 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/b05097add6e849a584b8db900ca1aa4e, entries=250, sequenceid=165, filesize=38.9 K 2023-04-18 22:54:22,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/c362bbfc5ac64e7b93ae5f5ecb9bc380 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/c362bbfc5ac64e7b93ae5f5ecb9bc380 2023-04-18 22:54:22,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/c362bbfc5ac64e7b93ae5f5ecb9bc380, entries=150, sequenceid=165, filesize=11.8 K 2023-04-18 22:54:22,476 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/c993cf7cd86342d1b51bee89ea820a4a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c993cf7cd86342d1b51bee89ea820a4a 2023-04-18 22:54:22,479 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:22,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:22,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:22,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:22,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:22,484 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:22,484 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:22,484 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:22,484 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:22,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,485 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c993cf7cd86342d1b51bee89ea820a4a, entries=150, sequenceid=165, filesize=11.8 K 2023-04-18 22:54:22,486 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:22,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:22,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:22,486 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,486 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 052b0c71a65b8786b7851f4ab1dab728 in 561ms, sequenceid=165, compaction requested=true 2023-04-18 22:54:22,486 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:22,487 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:22,487 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,487 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,487 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:22,487 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:22,487 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-18 22:54:22,487 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:22,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:22,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:22,488 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:22,488 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/A is initiating minor compaction (all files) 2023-04-18 22:54:22,488 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36096 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:22,488 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/A in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:22,488 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/B is initiating minor compaction (all files) 2023-04-18 22:54:22,489 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/060aa7cdbdc94d428e4c284d7bb9ff28, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/a06d5ad76ac54652be3830b9eb108fbb, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/b05097add6e849a584b8db900ca1aa4e] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=89.5 K 2023-04-18 22:54:22,489 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/B in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:22,489 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/01a57e5e38544c68bd248124df3c5e2c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b5cd34dc3c3c41bcb5be3cbc442feddd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/c362bbfc5ac64e7b93ae5f5ecb9bc380] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=35.3 K 2023-04-18 22:54:22,489 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 060aa7cdbdc94d428e4c284d7bb9ff28, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681858461130 2023-04-18 22:54:22,489 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 01a57e5e38544c68bd248124df3c5e2c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681858461130 2023-04-18 22:54:22,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:22,491 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting a06d5ad76ac54652be3830b9eb108fbb, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681858461215 2023-04-18 22:54:22,491 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting b5cd34dc3c3c41bcb5be3cbc442feddd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681858461215 2023-04-18 22:54:22,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:22,491 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:22,491 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:22,491 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:22,491 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting b05097add6e849a584b8db900ca1aa4e, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1681858461804 2023-04-18 22:54:22,491 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:22,491 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:22,491 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting c362bbfc5ac64e7b93ae5f5ecb9bc380, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1681858461816 2023-04-18 22:54:22,491 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:22,492 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:22,492 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-18 22:54:22,492 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:22,493 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:22,493 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:22,494 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:22,494 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:22,494 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:22,509 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#B#compaction#149 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:22,538 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#A#compaction#150 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:22,573 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304184974da5b8bd642c6af1c9c8692c80e64 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304184974da5b8bd642c6af1c9c8692c80e64 2023-04-18 22:54:22,577 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/5e2f70206e5c422596f05119d7b3f2ae as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/5e2f70206e5c422596f05119d7b3f2ae 2023-04-18 22:54:22,585 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/B of 052b0c71a65b8786b7851f4ab1dab728 into 5e2f70206e5c422596f05119d7b3f2ae(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:22,585 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:22,585 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/B, priority=13, startTime=1681858462487; duration=0sec 2023-04-18 22:54:22,585 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:22,586 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:22,587 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36096 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:22,587 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/C is initiating minor compaction (all files) 2023-04-18 22:54:22,587 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/C in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:22,587 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/828e6b67ad614ec7b6c7e34e7213e1ec, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bb1c64134f12450dad52c348b90e06f2, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c993cf7cd86342d1b51bee89ea820a4a] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=35.3 K 2023-04-18 22:54:22,587 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 828e6b67ad614ec7b6c7e34e7213e1ec, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681858461130 2023-04-18 22:54:22,588 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting bb1c64134f12450dad52c348b90e06f2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681858461215 2023-04-18 22:54:22,588 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting c993cf7cd86342d1b51bee89ea820a4a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1681858461816 2023-04-18 22:54:22,596 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#C#compaction#152 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:22,613 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/169809fbf04a432090629136ee072d0b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/169809fbf04a432090629136ee072d0b 2023-04-18 22:54:22,630 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/C of 052b0c71a65b8786b7851f4ab1dab728 into 169809fbf04a432090629136ee072d0b(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:22,630 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:22,630 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/C, priority=13, startTime=1681858462487; duration=0sec 2023-04-18 22:54:22,630 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:22,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:22,664 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:22,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:22,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858522671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:22,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858522673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:22,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858522774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:22,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858522775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:22,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858522979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:22,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858522979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:22,981 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=200, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/b4c33416b8b64756a9bac1f3c604779b 2023-04-18 22:54:23,002 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/83b37d833940414aa1ba0a859242ce09 2023-04-18 22:54:23,007 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/9de05ddfb9b443b6ae95d5fc0c2c0865 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/9de05ddfb9b443b6ae95d5fc0c2c0865 2023-04-18 22:54:23,012 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/A of 052b0c71a65b8786b7851f4ab1dab728 into 9de05ddfb9b443b6ae95d5fc0c2c0865(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:23,012 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:23,012 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/A, priority=13, startTime=1681858462486; duration=0sec 2023-04-18 22:54:23,012 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:23,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:23,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858523281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:23,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858523283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:23,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858523347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:23,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858523352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:23,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858523355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,415 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/d08ffd7f230448719633aad6388f49e6 2023-04-18 22:54:23,420 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/b4c33416b8b64756a9bac1f3c604779b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/b4c33416b8b64756a9bac1f3c604779b 2023-04-18 22:54:23,434 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/b4c33416b8b64756a9bac1f3c604779b, entries=150, sequenceid=200, filesize=25.3 K 2023-04-18 22:54:23,435 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/83b37d833940414aa1ba0a859242ce09 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/83b37d833940414aa1ba0a859242ce09 2023-04-18 22:54:23,450 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/83b37d833940414aa1ba0a859242ce09, entries=150, sequenceid=200, filesize=11.8 K 2023-04-18 22:54:23,451 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/d08ffd7f230448719633aad6388f49e6 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/d08ffd7f230448719633aad6388f49e6 2023-04-18 22:54:23,463 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/d08ffd7f230448719633aad6388f49e6, entries=150, sequenceid=200, filesize=11.8 K 2023-04-18 22:54:23,464 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 052b0c71a65b8786b7851f4ab1dab728 in 972ms, sequenceid=200, compaction requested=false 2023-04-18 22:54:23,464 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:23,464 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:23,464 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:23,464 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:23,464 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,465 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:23,465 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:23,466 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:23,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:23,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ddcda2a[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:23,469 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:23,469 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:23,469 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:23,469 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:23,470 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:23,471 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:23,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:23,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:23,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:23,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:23,471 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,471 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:23,473 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:23,474 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:23,474 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:23,474 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:23,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:23,477 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:23,477 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:23,477 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:23,477 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:23,477 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:23,479 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:23,479 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:23,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:23,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:23,480 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:23,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:23,480 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:23,482 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,485 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,485 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:23,486 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:23,486 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,486 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:23,486 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:23,486 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:23,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:23,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:23,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:23,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:23,485 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:23,486 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:23,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:23,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:23,486 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:23,486 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:23,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:23,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:23,586 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:23,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:23,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:23,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:23,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:23,591 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:23,591 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:23,591 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:23,591 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:23,591 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:23,591 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:23,593 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:23,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:23,593 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,593 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,593 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:23,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:23,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:23,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:23,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:23,595 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:23,595 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:23,595 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:23,595 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:23,595 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:23,595 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:23,595 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:23,595 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-18 22:54:23,596 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:23,596 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:23,596 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:23,596 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:23,596 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:23,596 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:23,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:23,791 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:23,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858463810, max timestamp=1681858463825Num uniques 859; 2023-04-18 22:54:23,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:23,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858463810, max timestamp=1681858463825Num uniques 859; 2023-04-18 22:54:23,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:23,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858463810, max timestamp=1681858463825Num uniques 859; 2023-04-18 22:54:23,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:23,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:23,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858523854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:23,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858523854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:23,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:23,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858523961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:23,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858523961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,043 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041891be7686d08746d6994ef3a89576b6af to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041891be7686d08746d6994ef3a89576b6af 2023-04-18 22:54:24,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:24,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858524164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:24,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858524164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,449 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=214, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/7db9fc67b08a48b788099975584db219 2023-04-18 22:54:24,461 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/60c1e18f41e845249624c924f50c5351 2023-04-18 22:54:24,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:24,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59006 deadline: 1681858524468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:24,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58994 deadline: 1681858524469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,485 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/16f52ded90ed4a3eaafd8d8d751890e4 2023-04-18 22:54:24,490 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/7db9fc67b08a48b788099975584db219 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/7db9fc67b08a48b788099975584db219 2023-04-18 22:54:24,494 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/7db9fc67b08a48b788099975584db219, entries=100, sequenceid=214, filesize=18.5 K 2023-04-18 22:54:24,496 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/60c1e18f41e845249624c924f50c5351 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/60c1e18f41e845249624c924f50c5351 2023-04-18 22:54:24,500 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/60c1e18f41e845249624c924f50c5351, entries=100, sequenceid=214, filesize=9.4 K 2023-04-18 22:54:24,501 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/16f52ded90ed4a3eaafd8d8d751890e4 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/16f52ded90ed4a3eaafd8d8d751890e4 2023-04-18 22:54:24,505 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/16f52ded90ed4a3eaafd8d8d751890e4, entries=100, sequenceid=214, filesize=9.4 K 2023-04-18 22:54:24,505 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 052b0c71a65b8786b7851f4ab1dab728 in 910ms, sequenceid=214, compaction requested=true 2023-04-18 22:54:24,506 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:24,506 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:24,506 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:24,506 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:24,506 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,506 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:24,506 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:24,511 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,511 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:24,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:24,511 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,511 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:24,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:24,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:24,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:24,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:24,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@391dd075[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:24,513 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:24,513 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,514 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,514 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:24,514 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:24,514 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:24,514 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,514 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:24,516 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:24,516 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,516 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:24,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:24,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:24,516 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:24,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:24,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:24,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:24,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:24,518 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:24,518 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:24,518 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:24,518 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:24,518 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:24,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:24,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:24,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:24,522 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,523 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:24,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:24,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:24,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:24,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:24,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:24,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:24,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,526 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:24,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:24,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:24,527 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:24,527 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:24,527 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:24,627 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:24,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:24,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:24,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:24,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:24,630 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:24,630 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:24,630 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:24,630 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:24,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:24,632 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:24,632 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:24,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:24,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:24,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:24,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:24,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:24,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:24,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:24,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:24,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:24,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:24,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:24,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:24,634 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:24,634 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:24,634 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-18 22:54:24,635 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:24,635 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:24,635 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:24,635 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:24,635 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:24,635 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:24,658 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418c26a61658f8145bc8755ddd2328bd00c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c26a61658f8145bc8755ddd2328bd00c 2023-04-18 22:54:24,666 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=249, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/664ab986fdf44063a541a43319ba4b44 2023-04-18 22:54:24,682 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/637397546b294984a78a100becc09692 2023-04-18 22:54:24,696 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/99b4343382ed4895802d70bde55e496d 2023-04-18 22:54:24,702 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/664ab986fdf44063a541a43319ba4b44 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/664ab986fdf44063a541a43319ba4b44 2023-04-18 22:54:24,720 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/664ab986fdf44063a541a43319ba4b44, entries=150, sequenceid=249, filesize=25.3 K 2023-04-18 22:54:24,721 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/637397546b294984a78a100becc09692 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/637397546b294984a78a100becc09692 2023-04-18 22:54:24,725 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/637397546b294984a78a100becc09692, entries=150, sequenceid=249, filesize=11.8 K 2023-04-18 22:54:24,726 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/99b4343382ed4895802d70bde55e496d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/99b4343382ed4895802d70bde55e496d 2023-04-18 22:54:24,730 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/99b4343382ed4895802d70bde55e496d, entries=150, sequenceid=249, filesize=11.8 K 2023-04-18 22:54:24,731 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for 052b0c71a65b8786b7851f4ab1dab728 in 97ms, sequenceid=249, compaction requested=true 2023-04-18 22:54:24,731 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:24,732 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:24,732 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:24,732 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:24,732 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,732 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:24,732 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:24,733 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,733 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:24,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:24,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:24,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@249fb8dc[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:24,736 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:24,736 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,738 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,738 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:24,738 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:24,738 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,738 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:24,738 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:24,740 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:24,740 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,740 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:24,740 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:24,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:24,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:24,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:24,743 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:24,743 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:24,743 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:24,743 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:24,743 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:24,745 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,745 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,745 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:24,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:24,745 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:24,747 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:24,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,749 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,749 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:24,749 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:24,749 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:24,749 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:24,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:24,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:24,749 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,750 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,750 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:24,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:24,750 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:24,749 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:24,750 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:24,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:24,750 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:24,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:24,850 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:24,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:24,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:24,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:24,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:24,854 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:24,854 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:24,854 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:24,854 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:24,855 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,855 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:24,857 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:24,857 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:24,858 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,858 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:24,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:24,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:24,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:24,859 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:24,859 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:24,859 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:24,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:24,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:24,860 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:24,860 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:24,860 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:24,860 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:24,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:24,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:24,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:24,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:24,862 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,862 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:24,862 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,862 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@243fd088[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:24,864 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:24,864 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,865 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:24,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:24,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:24,866 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,866 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:24,867 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:24,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:24,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:24,870 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:24,870 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:24,870 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:24,870 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:24,870 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:24,873 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,873 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,893 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:24,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:24,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:24,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:24,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:24,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:24,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,894 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:24,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,895 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:24,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:24,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,898 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:24,898 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:24,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:24,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:24,898 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:24,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:24,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:24,899 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:24,899 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:24,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:24,899 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:24,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:24,899 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:24,899 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:24,899 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,899 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:24,899 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:24,899 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:24,999 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:25,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:25,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:25,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:25,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:25,004 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:25,004 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:25,004 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:25,004 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:25,004 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:25,004 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:25,006 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,006 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:25,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:25,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:25,007 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,007 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:25,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:25,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:25,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:25,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:25,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:25,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:25,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:25,016 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:25,016 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:25,016 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:25,016 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:25,016 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-18 22:54:25,016 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:25,017 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:25,017 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:25,017 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:25,017 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:25,017 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:25,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:25,020 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:25,044 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418dab5d1ecc8ff43988bb4ed083538e566 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418dab5d1ecc8ff43988bb4ed083538e566 2023-04-18 22:54:25,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858465036, max timestamp=1681858465066Num uniques 859; 2023-04-18 22:54:25,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858465036, max timestamp=1681858465066Num uniques 859; 2023-04-18 22:54:25,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:54:25,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:54:25,070 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=258, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/ca5f5c43565044ceaf4c08937d591d3a 2023-04-18 22:54:25,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858465036, max timestamp=1681858465066Num uniques 859; 2023-04-18 22:54:25,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:54:25,097 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/a405e11016504bf9924ce7e5c4d7c945 2023-04-18 22:54:25,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:25,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858525112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:25,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858525117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:25,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858525218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:25,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858525221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:25,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858525356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:25,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858525359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,362 DEBUG [Thread-984] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:54:25,362 DEBUG [Thread-982] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:54:25,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:25,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858525365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,369 DEBUG [Thread-976] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:54:25,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:25,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858525420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:25,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858525426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,518 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/7910f6b3d4bc4af4b4200b58c54cee3d 2023-04-18 22:54:25,523 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/ca5f5c43565044ceaf4c08937d591d3a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/ca5f5c43565044ceaf4c08937d591d3a 2023-04-18 22:54:25,528 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/ca5f5c43565044ceaf4c08937d591d3a, entries=250, sequenceid=258, filesize=39.0 K 2023-04-18 22:54:25,529 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/a405e11016504bf9924ce7e5c4d7c945 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/a405e11016504bf9924ce7e5c4d7c945 2023-04-18 22:54:25,533 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/a405e11016504bf9924ce7e5c4d7c945, entries=150, sequenceid=258, filesize=11.8 K 2023-04-18 22:54:25,534 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/7910f6b3d4bc4af4b4200b58c54cee3d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/7910f6b3d4bc4af4b4200b58c54cee3d 2023-04-18 22:54:25,538 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/7910f6b3d4bc4af4b4200b58c54cee3d, entries=150, sequenceid=258, filesize=11.8 K 2023-04-18 22:54:25,540 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 052b0c71a65b8786b7851f4ab1dab728 in 524ms, sequenceid=258, compaction requested=true 2023-04-18 22:54:25,540 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:25,540 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:25,542 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,542 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:25,542 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:25,543 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:25,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:25,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:25,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:25,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:25,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:25,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f3943af[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:25,544 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:25,544 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:25,551 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:25,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:25,552 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:25,552 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:25,552 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:25,552 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,552 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:25,553 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:25,553 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,553 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:25,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:25,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:25,553 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:25,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:25,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:25,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:25,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:25,556 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:25,556 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:25,556 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:25,556 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:25,556 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:25,558 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:25,558 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:25,558 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:25,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:25,559 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:25,560 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:25,562 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,562 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:25,562 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:25,563 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:25,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:25,562 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:25,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:25,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:25,563 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:25,563 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,563 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:25,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:25,563 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:25,563 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:25,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:25,563 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:25,563 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:25,663 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:25,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:25,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:25,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:25,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:25,667 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:25,668 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:25,668 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:25,668 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:25,668 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:25,668 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:25,671 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:25,671 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:25,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:25,671 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:25,671 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:25,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:25,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:25,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:25,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:25,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:25,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:25,673 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:25,673 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:25,673 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:25,673 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:25,673 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:25,673 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:25,673 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:25,673 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-18 22:54:25,673 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:25,673 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:25,674 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:25,674 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:25,674 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:25,674 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:25,688 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418b59522e098a84a359766c2bac00542d4 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418b59522e098a84a359766c2bac00542d4 2023-04-18 22:54:25,694 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=295, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/af2ce903d08d4883af4505ae61611ae5 2023-04-18 22:54:25,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:25,730 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:25,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:25,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858525763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:25,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858525763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:25,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858525868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:25,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:25,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858525868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:26,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858526073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:26,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858526073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,106 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/d401afac6cd947f28c166c183728ab1e 2023-04-18 22:54:26,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:26,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858526376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:26,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858526380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,520 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/e567d7dc1adf44a7bdc78106dd9730a5 2023-04-18 22:54:26,525 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/af2ce903d08d4883af4505ae61611ae5 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/af2ce903d08d4883af4505ae61611ae5 2023-04-18 22:54:26,530 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/af2ce903d08d4883af4505ae61611ae5, entries=150, sequenceid=295, filesize=25.4 K 2023-04-18 22:54:26,531 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/d401afac6cd947f28c166c183728ab1e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/d401afac6cd947f28c166c183728ab1e 2023-04-18 22:54:26,535 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/d401afac6cd947f28c166c183728ab1e, entries=150, sequenceid=295, filesize=11.9 K 2023-04-18 22:54:26,536 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/e567d7dc1adf44a7bdc78106dd9730a5 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e567d7dc1adf44a7bdc78106dd9730a5 2023-04-18 22:54:26,540 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e567d7dc1adf44a7bdc78106dd9730a5, entries=150, sequenceid=295, filesize=11.9 K 2023-04-18 22:54:26,541 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for 052b0c71a65b8786b7851f4ab1dab728 in 868ms, sequenceid=295, compaction requested=true 2023-04-18 22:54:26,542 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:26,542 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:26,542 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:26,542 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:26,542 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,542 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:26,542 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:26,544 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,544 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:26,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:26,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:26,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:26,544 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:26,544 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:26,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:26,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:26,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:26,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:26,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7870b045[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:26,546 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:26,546 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:26,547 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:26,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:26,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:26,547 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,547 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:26,549 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,549 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:26,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:26,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:26,549 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:26,549 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:26,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:26,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:26,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:26,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:26,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:26,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:26,551 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:26,551 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:26,551 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:26,551 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:26,551 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:26,553 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:26,554 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:26,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:26,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:26,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:26,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:26,554 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:26,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:26,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:26,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:26,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:26,555 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,556 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:26,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:26,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,560 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,560 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:26,560 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:26,560 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,560 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:26,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:26,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:26,560 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:26,560 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:26,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:26,560 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:26,560 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:26,560 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:26,560 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:26,560 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:26,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:26,660 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:26,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:26,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:26,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:26,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:26,665 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:26,665 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:26,665 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:26,665 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:26,665 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:26,665 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:26,668 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,668 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:26,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:26,668 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:26,668 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:26,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:26,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:26,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:26,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:26,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:26,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:26,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:26,670 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:26,670 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:26,670 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:26,670 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:26,670 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:26,670 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:26,670 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:26,670 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-18 22:54:26,671 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:26,671 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:26,671 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:26,671 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:26,671 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:26,671 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:26,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:26,893 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:26,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858466911, max timestamp=1681858466921Num uniques 859; 2023-04-18 22:54:26,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:54:26,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858466911, max timestamp=1681858466921Num uniques 859; 2023-04-18 22:54:26,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858466911, max timestamp=1681858466921Num uniques 859; 2023-04-18 22:54:26,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:54:26,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:54:26,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681858466949, max timestamp=1681858466963Num uniques 1018; 2023-04-18 22:54:26,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681858466949, max timestamp=1681858466963Num uniques 1018; 2023-04-18 22:54:26,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-18 22:54:26,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-18 22:54:26,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681858466949, max timestamp=1681858466963Num uniques 1018; 2023-04-18 22:54:26,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-18 22:54:27,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681858466993, max timestamp=1681858467008Num uniques 1077; 2023-04-18 22:54:27,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-18 22:54:27,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681858466993, max timestamp=1681858467008Num uniques 1077; 2023-04-18 22:54:27,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-18 22:54:27,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681858466993, max timestamp=1681858467008Num uniques 1077; 2023-04-18 22:54:27,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-18 22:54:27,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:27,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858527048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:27,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858527056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,087 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418df56147ba7e6490bb751967d8fcd7702 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418df56147ba7e6490bb751967d8fcd7702 2023-04-18 22:54:27,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:27,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858527157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:27,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858527164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:27,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858527362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:27,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858527366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,493 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=303, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/d37dad4c26964df2909add01b7a32b48 2023-04-18 22:54:27,508 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/090ba75ce0934790acdaa6b3b5f93a1d 2023-04-18 22:54:27,526 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/c8013ea5800548dcaf94b4905e9489d7 2023-04-18 22:54:27,533 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/d37dad4c26964df2909add01b7a32b48 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/d37dad4c26964df2909add01b7a32b48 2023-04-18 22:54:27,538 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/d37dad4c26964df2909add01b7a32b48, entries=150, sequenceid=303, filesize=25.4 K 2023-04-18 22:54:27,539 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/090ba75ce0934790acdaa6b3b5f93a1d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/090ba75ce0934790acdaa6b3b5f93a1d 2023-04-18 22:54:27,544 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/090ba75ce0934790acdaa6b3b5f93a1d, entries=150, sequenceid=303, filesize=11.9 K 2023-04-18 22:54:27,545 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/c8013ea5800548dcaf94b4905e9489d7 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c8013ea5800548dcaf94b4905e9489d7 2023-04-18 22:54:27,549 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c8013ea5800548dcaf94b4905e9489d7, entries=150, sequenceid=303, filesize=11.9 K 2023-04-18 22:54:27,550 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 052b0c71a65b8786b7851f4ab1dab728 in 880ms, sequenceid=303, compaction requested=true 2023-04-18 22:54:27,550 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:27,550 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:27,550 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:27,550 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:27,550 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,550 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:27,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:27,552 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,552 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:27,553 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,553 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:27,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:27,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21f0becf[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:27,557 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:27,557 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,561 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,561 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:27,561 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:27,561 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,561 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:27,561 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:27,563 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,563 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:27,563 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:27,563 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:27,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:27,567 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:27,567 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:27,567 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:27,567 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:27,567 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:27,569 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,569 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:27,570 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:27,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:27,570 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,572 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:27,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,574 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,574 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:27,574 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:27,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:27,575 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:27,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:27,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:27,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:27,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:27,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:27,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,575 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:27,575 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:27,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:27,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681858467039, max timestamp=1681858467671Num uniques 1136; 2023-04-18 22:54:27,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-18 22:54:27,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:27,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681858467039, max timestamp=1681858467671Num uniques 1136; 2023-04-18 22:54:27,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-18 22:54:27,674 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=111.82 KB heapSize=310.83 KB 2023-04-18 22:54:27,675 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:27,675 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:27,675 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:27,675 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:27,675 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:27,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681858467039, max timestamp=1681858467671Num uniques 1136; 2023-04-18 22:54:27,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-18 22:54:27,675 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:27,676 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:27,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:27,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:27,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:27,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:27,680 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:27,680 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:27,680 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:27,680 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:27,681 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,681 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:27,683 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,683 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:27,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:27,683 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,683 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:27,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:27,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:27,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:27,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:27,688 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:27,688 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:27,688 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:27,690 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:27,690 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:27,690 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:27,690 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:27,690 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:27,690 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:27,690 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:27,690 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:27,690 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,690 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:27,690 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:27,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,692 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:27,692 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,693 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:27,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:27,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b1181e9[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:27,695 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:27,695 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,697 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:27,697 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:27,697 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:27,697 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,697 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:27,698 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,698 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:27,698 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:27,699 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:27,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:27,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:27,701 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:27,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:27,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:27,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:27,701 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:27,706 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,706 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:27,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,707 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:27,708 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,710 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,712 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:27,712 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,712 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:27,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:27,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:27,712 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:27,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:27,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:27,712 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:27,713 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:27,713 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,713 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,713 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,713 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,713 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:27,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:27,713 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:27,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:27,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:27,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858467706, max timestamp=1681858467725Num uniques 859; 2023-04-18 22:54:27,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858467706, max timestamp=1681858467725Num uniques 859; 2023-04-18 22:54:27,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:27,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858467706, max timestamp=1681858467725Num uniques 859; 2023-04-18 22:54:27,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:27,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:27,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858467760, max timestamp=1681858467778Num uniques 1068; 2023-04-18 22:54:27,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-18 22:54:27,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858467760, max timestamp=1681858467778Num uniques 1068; 2023-04-18 22:54:27,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-18 22:54:27,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858467760, max timestamp=1681858467778Num uniques 1068; 2023-04-18 22:54:27,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-18 22:54:27,813 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:27,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:27,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:27,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:27,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:27,817 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:27,817 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:27,817 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:27,817 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:27,817 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,817 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:27,819 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:27,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:27,819 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:27,819 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,819 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:27,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:27,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:27,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:27,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:27,820 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:27,820 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:27,820 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:27,820 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:27,820 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:27,820 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:27,820 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:27,820 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:27,821 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:27,822 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,822 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:27,823 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,823 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:27,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:27,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e50811f[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:27,828 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:27,828 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,830 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,831 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:27,831 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:27,831 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:27,831 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,831 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:27,833 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:27,833 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,833 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:27,833 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:27,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:27,835 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:27,835 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:27,835 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:27,835 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:27,835 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:27,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681858467790, max timestamp=1681858467835Num uniques 1027; 2023-04-18 22:54:27,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-18 22:54:27,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681858467790, max timestamp=1681858467835Num uniques 1027; 2023-04-18 22:54:27,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-18 22:54:27,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681858467790, max timestamp=1681858467835Num uniques 1027; 2023-04-18 22:54:27,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-18 22:54:27,839 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,839 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,839 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:27,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:27,839 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:27,841 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,844 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:27,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:27,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:27,844 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:27,844 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:27,844 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:27,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:27,844 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:27,844 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:27,844 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:27,844 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,844 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,844 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:27,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:27,844 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:27,844 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:27,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681858467867, max timestamp=1681858467888Num uniques 1086; 2023-04-18 22:54:27,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-18 22:54:27,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681858467867, max timestamp=1681858467888Num uniques 1086; 2023-04-18 22:54:27,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-18 22:54:27,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681858467867, max timestamp=1681858467888Num uniques 1086; 2023-04-18 22:54:27,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-18 22:54:27,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681858467907, max timestamp=1681858467926Num uniques 1195; 2023-04-18 22:54:27,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=13510, suffix heap size=152248,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=295 2023-04-18 22:54:27,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681858467907, max timestamp=1681858467926Num uniques 1195; 2023-04-18 22:54:27,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=13510, suffix heap size=152248,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=295 2023-04-18 22:54:27,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681858467907, max timestamp=1681858467926Num uniques 1195; 2023-04-18 22:54:27,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=13510, suffix heap size=152248,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=295 2023-04-18 22:54:27,945 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:27,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:27,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:27,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:27,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:27,948 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:27,948 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:27,948 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:27,948 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:27,949 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,949 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:27,951 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:27,951 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:27,951 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,952 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:27,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:27,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:27,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:27,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:27,953 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:27,953 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:27,953 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:27,953 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:27,955 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,955 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:27,955 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,955 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:27,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:27,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@635ff808[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:27,957 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:27,957 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,958 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,958 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:27,958 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:27,958 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:27,958 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,958 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:27,959 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:27,959 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,959 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:27,960 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:27,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:27,963 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:27,964 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:27,964 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:27,964 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:27,964 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:27,967 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,967 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:27,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:27,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,967 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:27,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:27,967 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:27,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:27,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:27,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,969 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:27,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:27,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:27,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:27,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:27,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:27,972 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:27,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:27,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:27,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:27,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:27,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:27,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:27,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:27,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:27,972 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:27,972 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:27,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:27,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681858467950, max timestamp=1681858467974Num uniques 1154; 2023-04-18 22:54:27,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-18 22:54:27,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681858467950, max timestamp=1681858467974Num uniques 1154; 2023-04-18 22:54:27,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681858467950, max timestamp=1681858467974Num uniques 1154; 2023-04-18 22:54:27,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-18 22:54:27,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-18 22:54:28,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681858467989, max timestamp=1681858468014Num uniques 1113; 2023-04-18 22:54:28,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-18 22:54:28,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681858467989, max timestamp=1681858468014Num uniques 1113; 2023-04-18 22:54:28,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-18 22:54:28,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681858467989, max timestamp=1681858468014Num uniques 1113; 2023-04-18 22:54:28,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-18 22:54:28,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1681858468032, max timestamp=1681858468057Num uniques 1072; 2023-04-18 22:54:28,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1681858468032, max timestamp=1681858468057Num uniques 1072; 2023-04-18 22:54:28,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=14742, suffix heap size=139456,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=322 2023-04-18 22:54:28,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1681858468032, max timestamp=1681858468057Num uniques 1072; 2023-04-18 22:54:28,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=14742, suffix heap size=139456,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=322 2023-04-18 22:54:28,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=14742, suffix heap size=139456,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=322 2023-04-18 22:54:28,073 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:28,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:28,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:28,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:28,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:28,076 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:28,076 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:28,076 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:28,076 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:28,077 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,077 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,078 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:28,078 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:28,078 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,078 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:28,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:28,079 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:28,079 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:28,079 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:28,080 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:28,080 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:28,080 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:28,080 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:28,080 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:28,080 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:28,080 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:28,080 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:28,080 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,080 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:28,080 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:28,082 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,082 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,082 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,082 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:28,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@477d346a[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:28,083 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:28,083 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,088 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,088 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:28,088 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:28,088 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:28,088 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,088 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:28,089 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,090 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:28,090 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:28,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,090 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:28,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:28,092 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:28,092 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:28,092 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:28,092 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:28,092 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:28,094 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,094 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,094 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:28,094 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,096 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,098 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418c27c5d1d519547d4b42c915c2313722a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c27c5d1d519547d4b42c915c2313722a 2023-04-18 22:54:28,102 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,102 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:28,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:28,102 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,102 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,102 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:28,103 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:28,103 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:28,103 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,103 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:28,103 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:28,103 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:28,103 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:28,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:28,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:28,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858528103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:28,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858528104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,203 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:28,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:28,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:28,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:28,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:28,208 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:28,208 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:28,208 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:28,208 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:28,209 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,209 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,210 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,210 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:28,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:28,210 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,210 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:28,211 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:28,211 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:28,212 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:28,214 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:28,214 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:28,214 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:28,214 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:28,214 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:28,214 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:28,214 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:28,214 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:28,214 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,214 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:28,214 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:28,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:28,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858528213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,216 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,216 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858528213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:28,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3014eba5[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:28,218 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:28,218 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,219 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,220 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,220 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:28,229 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,229 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:28,229 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:28,229 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:28,229 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:28,231 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:28,232 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:28,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:28,234 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:28,234 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:28,235 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:28,235 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:28,235 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:28,237 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,237 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,237 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:28,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,238 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:28,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:28,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,239 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,242 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:28,242 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:28,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:28,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:28,242 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,242 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,242 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:28,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:28,242 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,243 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,243 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,243 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,243 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:28,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:28,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:28,243 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:28,243 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:28,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:28,343 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:28,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:28,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:28,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:28,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:28,347 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:28,347 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:28,347 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:28,347 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:28,347 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,347 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,349 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,349 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:28,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:28,349 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,349 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:28,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:28,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:28,351 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:28,351 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:28,351 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:28,351 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:28,351 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:28,351 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:28,351 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:28,351 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:28,351 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:28,351 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:28,351 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:28,351 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,351 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:28,351 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:28,354 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,354 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,354 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,354 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:28,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a56ccc1[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:28,356 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:28,356 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,357 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:28,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:28,357 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,357 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:28,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:28,359 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,359 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:28,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,359 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:28,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,359 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:28,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:28,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:28,362 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:28,362 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:28,362 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:28,362 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:28,362 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:28,364 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,364 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,364 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:28,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,365 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,366 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:28,369 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:28,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:28,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:28,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:28,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:28,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:28,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:28,370 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,370 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,370 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,370 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:28,370 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:28,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:28,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:28,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858528416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:28,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858528418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,470 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:28,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:28,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:28,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:28,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:28,478 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:28,478 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:28,478 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:28,478 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:28,479 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,479 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,480 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,480 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:28,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:28,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:28,480 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,480 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:28,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:28,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:28,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:28,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:28,481 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:28,481 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:28,481 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:28,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:28,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:28,482 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:28,482 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:28,482 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:28,482 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:28,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:28,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:28,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:28,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:28,485 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:28,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3655ee3b[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:28,486 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:28,487 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,488 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,488 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:28,488 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:28,488 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:28,488 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,488 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:28,490 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,490 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:28,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,490 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:28,490 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:28,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:28,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:28,493 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:28,493 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:28,493 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:28,493 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:28,493 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:28,494 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,494 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,495 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,496 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,501 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:28,501 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:28,501 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:28,501 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:28,501 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:28,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:28,501 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,501 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,502 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:28,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:28,502 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,502 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:28,502 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:28,502 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:28,506 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=376, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/cd1d4fdf3ef84db9a1f5863316b8707b 2023-04-18 22:54:28,602 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:28,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:28,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:28,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:28,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:28,606 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:28,606 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:28,606 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:28,606 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:28,606 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,606 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,608 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:28,608 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:28,609 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,609 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:28,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:28,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:28,610 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:28,610 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:28,610 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:28,610 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:28,610 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:28,610 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:28,610 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:28,610 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:28,610 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:28,610 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:28,610 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:28,611 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,611 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:28,611 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:28,612 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,612 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,613 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,613 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:28,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:28,614 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:28,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@182542ff[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:28,614 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,615 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,616 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:28,616 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:28,616 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,616 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:28,616 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:28,617 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,617 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:28,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,617 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:28,617 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:28,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:28,620 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:28,620 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:28,620 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:28,620 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:28,620 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:28,623 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,623 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,623 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,623 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,627 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:28,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:28,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:28,627 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:28,627 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:28,628 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,628 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,628 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,628 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,628 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,628 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,628 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,632 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:28,632 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:28,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:28,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:28,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:28,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858528723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,728 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:28,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:28,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858528725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:28,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:28,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:28,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:28,731 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:28,731 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:28,731 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:28,731 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:28,731 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,731 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,733 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,733 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:28,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:28,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:28,733 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,733 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:28,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:28,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:28,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:28,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:28,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:28,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:28,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:28,735 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:28,735 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:28,735 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:28,735 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:28,735 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:28,735 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:28,735 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:28,735 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:28,735 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,735 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:28,735 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:28,736 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,736 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,737 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,737 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:28,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:28,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ffbcb9d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:28,738 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:28,738 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,741 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,741 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:28,741 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:28,741 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:28,741 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,741 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:28,742 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:28,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:28,745 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:28,745 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:28,745 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:28,745 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:28,745 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:28,746 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,747 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,747 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:28,747 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,749 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,751 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,751 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,751 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,751 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:28,751 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:28,751 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:28,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:28,751 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:28,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:28,751 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:28,751 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,752 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:28,752 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:28,752 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:28,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:28,852 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:28,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:28,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:28,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:28,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:28,855 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:28,855 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:28,855 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:28,855 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:28,856 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,856 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,858 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,858 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:28,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:28,858 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,858 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:28,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:28,859 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:28,859 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:28,859 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:28,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:28,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:28,860 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:28,860 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:28,860 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:28,860 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:28,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:28,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:28,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:28,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:28,861 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,862 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,862 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,862 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:28,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:28,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13572e67[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:28,863 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:28,864 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,865 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,865 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:28,865 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,865 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:28,865 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:28,865 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:28,867 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:28,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:28,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:28,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:28,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:28,869 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:28,869 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:28,869 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:28,869 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:28,869 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:28,872 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,872 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,872 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:28,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:28,873 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:28,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,874 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,876 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,876 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:28,876 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:28,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:28,876 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:28,876 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:28,877 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:28,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:28,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:28,876 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,877 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,877 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,877 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,877 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,877 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:28,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:28,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:28,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:28,917 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/fe6b94933a664252ab331cd1350b2aef 2023-04-18 22:54:28,977 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:28,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:28,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:28,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:28,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:28,981 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:28,981 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:28,981 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:28,981 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:28,981 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,981 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,984 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:28,984 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:28,984 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,984 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:28,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:28,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:28,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:28,985 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:28,985 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:28,985 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:28,986 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:28,986 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:28,986 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:28,986 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:28,986 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:28,986 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:28,986 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:28,986 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:28,986 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,986 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:28,986 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:28,991 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,991 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,991 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,991 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:28,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a6a3e00[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:28,993 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:28,993 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,994 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:28,995 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:28,995 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,995 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:28,995 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:28,995 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:28,996 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:28,996 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,996 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:28,997 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:28,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:28,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:28,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:28,999 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:28,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:28,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:28,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:28,999 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:29,001 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,001 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,001 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:29,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,002 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,003 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,006 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,006 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,006 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,006 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,006 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,006 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,006 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,006 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:29,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:29,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:29,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,013 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:29,013 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:29,013 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:29,013 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:29,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:29,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:29,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:29,113 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:29,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:29,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:29,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:29,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:29,117 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:29,117 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:29,117 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:29,117 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:29,118 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,118 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,120 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:29,120 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:29,121 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,121 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:29,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:29,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:29,122 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:29,122 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:29,123 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:29,123 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:29,123 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:29,123 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:29,123 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:29,123 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,123 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:29,123 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:29,126 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,126 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,126 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,126 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:29,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@386ab708[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:29,128 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:29,128 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,129 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,130 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:29,130 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:29,130 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:29,130 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,130 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:29,132 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,132 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,132 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:29,132 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:29,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:29,135 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:29,135 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:29,135 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:29,135 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:29,135 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:29,137 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,137 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:29,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,137 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:29,137 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,138 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,141 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:29,141 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:29,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:29,141 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,141 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,141 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,141 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,141 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,141 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,141 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:29,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:29,141 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:29,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:29,142 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:29,142 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:29,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:29,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:29,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858529229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:29,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858529232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,242 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:29,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:29,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:29,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:29,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:29,247 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:29,247 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:29,247 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:29,247 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:29,248 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,248 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,249 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,249 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:29,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:29,249 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,249 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:29,251 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:29,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:29,251 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:29,251 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:29,251 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:29,251 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:29,252 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:29,252 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,252 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:29,252 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:29,256 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,257 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,257 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,257 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@699c34bb[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:29,258 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:29,259 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,260 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,260 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:29,260 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:29,260 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:29,260 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,261 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:29,262 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,262 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:29,262 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,262 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:29,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:29,265 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:29,265 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:29,265 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:29,265 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:29,265 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:29,267 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,267 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,267 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,267 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,268 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,271 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,271 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,271 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:29,272 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:29,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:29,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:29,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:29,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:29,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:29,272 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:29,272 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:29,328 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/bfa538804d1f4edc9cfc443d0d255646 2023-04-18 22:54:29,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/cd1d4fdf3ef84db9a1f5863316b8707b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/cd1d4fdf3ef84db9a1f5863316b8707b 2023-04-18 22:54:29,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/cd1d4fdf3ef84db9a1f5863316b8707b, entries=250, sequenceid=376, filesize=39.2 K 2023-04-18 22:54:29,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/fe6b94933a664252ab331cd1350b2aef as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/fe6b94933a664252ab331cd1350b2aef 2023-04-18 22:54:29,346 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/fe6b94933a664252ab331cd1350b2aef, entries=150, sequenceid=376, filesize=11.9 K 2023-04-18 22:54:29,347 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/bfa538804d1f4edc9cfc443d0d255646 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bfa538804d1f4edc9cfc443d0d255646 2023-04-18 22:54:29,351 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bfa538804d1f4edc9cfc443d0d255646, entries=150, sequenceid=376, filesize=11.9 K 2023-04-18 22:54:29,353 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~83.58 KB/85584, currentSize=154.31 KB/158010 for 052b0c71a65b8786b7851f4ab1dab728 in 1679ms, sequenceid=376, compaction requested=true 2023-04-18 22:54:29,353 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:29,353 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:29,353 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-18 22:54:29,353 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-18 22:54:29,353 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:29,354 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:29,357 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 229028 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-18 22:54:29,357 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/A is initiating minor compaction (all files) 2023-04-18 22:54:29,357 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/A in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:29,357 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/9de05ddfb9b443b6ae95d5fc0c2c0865, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/b4c33416b8b64756a9bac1f3c604779b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/7db9fc67b08a48b788099975584db219, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/664ab986fdf44063a541a43319ba4b44, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/ca5f5c43565044ceaf4c08937d591d3a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/af2ce903d08d4883af4505ae61611ae5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/d37dad4c26964df2909add01b7a32b48, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/cd1d4fdf3ef84db9a1f5863316b8707b] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=223.7 K 2023-04-18 22:54:29,358 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 9de05ddfb9b443b6ae95d5fc0c2c0865, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1681858461816 2023-04-18 22:54:29,358 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 94634 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-18 22:54:29,358 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/B is initiating minor compaction (all files) 2023-04-18 22:54:29,358 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/B in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:29,358 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/5e2f70206e5c422596f05119d7b3f2ae, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/83b37d833940414aa1ba0a859242ce09, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/60c1e18f41e845249624c924f50c5351, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/637397546b294984a78a100becc09692, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/a405e11016504bf9924ce7e5c4d7c945, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/d401afac6cd947f28c166c183728ab1e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/090ba75ce0934790acdaa6b3b5f93a1d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/fe6b94933a664252ab331cd1350b2aef] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=92.4 K 2023-04-18 22:54:29,359 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting b4c33416b8b64756a9bac1f3c604779b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1681858462016 2023-04-18 22:54:29,359 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 5e2f70206e5c422596f05119d7b3f2ae, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1681858461816 2023-04-18 22:54:29,359 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 7db9fc67b08a48b788099975584db219, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1681858462667 2023-04-18 22:54:29,360 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 83b37d833940414aa1ba0a859242ce09, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1681858462016 2023-04-18 22:54:29,360 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 664ab986fdf44063a541a43319ba4b44, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681858463849 2023-04-18 22:54:29,361 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 60c1e18f41e845249624c924f50c5351, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1681858462667 2023-04-18 22:54:29,361 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting ca5f5c43565044ceaf4c08937d591d3a, keycount=250, bloomtype=ROW, size=39.0 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1681858464974 2023-04-18 22:54:29,361 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 637397546b294984a78a100becc09692, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681858463849 2023-04-18 22:54:29,362 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting a405e11016504bf9924ce7e5c4d7c945, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1681858465004 2023-04-18 22:54:29,362 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting af2ce903d08d4883af4505ae61611ae5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1681858465096 2023-04-18 22:54:29,362 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting d401afac6cd947f28c166c183728ab1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1681858465096 2023-04-18 22:54:29,363 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting d37dad4c26964df2909add01b7a32b48, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681858465728 2023-04-18 22:54:29,363 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 090ba75ce0934790acdaa6b3b5f93a1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681858465728 2023-04-18 22:54:29,363 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting cd1d4fdf3ef84db9a1f5863316b8707b, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1681858467039 2023-04-18 22:54:29,363 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting fe6b94933a664252ab331cd1350b2aef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1681858467055 2023-04-18 22:54:29,372 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:29,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:29,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:29,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:29,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:29,384 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:29,384 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:29,384 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:29,384 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:29,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:29,385 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=161.02 KB heapSize=450.09 KB 2023-04-18 22:54:29,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681858468083, max timestamp=1681858469383Num uniques 1181; 2023-04-18 22:54:29,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=8286, suffix heap size=150792,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=181 2023-04-18 22:54:29,387 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,387 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681858468083, max timestamp=1681858469383Num uniques 1181; 2023-04-18 22:54:29,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=8286, suffix heap size=150792,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=181 2023-04-18 22:54:29,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681858468083, max timestamp=1681858469383Num uniques 1181; 2023-04-18 22:54:29,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=8286, suffix heap size=150792,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=181 2023-04-18 22:54:29,387 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:29,387 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:29,387 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:29,388 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:29,388 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:29,388 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:29,388 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:29,389 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,389 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,389 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:29,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:29,394 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#B#compaction#174 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:29,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:29,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:29,400 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:29,400 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:29,400 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:29,402 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:29,402 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:29,402 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:29,402 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:29,402 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:29,402 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:29,403 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:29,403 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:29,403 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,403 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:29,403 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:29,404 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,405 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,405 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,405 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:29,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:29,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43120bf9[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:29,408 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:29,408 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,414 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#A#compaction#173 average throughput is 0.76 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:29,416 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,416 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:29,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:29,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:29,418 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,418 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:29,419 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:29,419 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:29,419 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,419 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:29,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:29,423 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:29,423 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:29,423 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:29,423 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:29,423 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:29,425 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,425 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,425 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:29,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:29,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:29,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,435 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,435 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/24c3af227571410cb0838dfded867043 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/24c3af227571410cb0838dfded867043 2023-04-18 22:54:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,438 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,442 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:29,442 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:29,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:29,442 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,442 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,442 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,442 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,442 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,442 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,442 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,442 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:29,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:29,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:29,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:29,443 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:29,443 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:29,445 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041805f3e1caa2c445368dc4e7f8b9987a91 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041805f3e1caa2c445368dc4e7f8b9987a91 2023-04-18 22:54:29,448 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/B of 052b0c71a65b8786b7851f4ab1dab728 into 24c3af227571410cb0838dfded867043(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:29,449 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:29,449 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/B, priority=8, startTime=1681858469353; duration=0sec 2023-04-18 22:54:29,449 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:29,449 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-18 22:54:29,452 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 94634 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-18 22:54:29,452 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/C is initiating minor compaction (all files) 2023-04-18 22:54:29,452 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/C in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:29,452 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/169809fbf04a432090629136ee072d0b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/d08ffd7f230448719633aad6388f49e6, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/16f52ded90ed4a3eaafd8d8d751890e4, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/99b4343382ed4895802d70bde55e496d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/7910f6b3d4bc4af4b4200b58c54cee3d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e567d7dc1adf44a7bdc78106dd9730a5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c8013ea5800548dcaf94b4905e9489d7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bfa538804d1f4edc9cfc443d0d255646] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=92.4 K 2023-04-18 22:54:29,453 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 169809fbf04a432090629136ee072d0b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1681858461816 2023-04-18 22:54:29,453 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting d08ffd7f230448719633aad6388f49e6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1681858462016 2023-04-18 22:54:29,453 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 16f52ded90ed4a3eaafd8d8d751890e4, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1681858462667 2023-04-18 22:54:29,454 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 99b4343382ed4895802d70bde55e496d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681858463849 2023-04-18 22:54:29,454 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 7910f6b3d4bc4af4b4200b58c54cee3d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1681858465004 2023-04-18 22:54:29,455 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting e567d7dc1adf44a7bdc78106dd9730a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1681858465096 2023-04-18 22:54:29,455 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting c8013ea5800548dcaf94b4905e9489d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681858465728 2023-04-18 22:54:29,456 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting bfa538804d1f4edc9cfc443d0d255646, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1681858467055 2023-04-18 22:54:29,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858469396, max timestamp=1681858469455Num uniques 859; 2023-04-18 22:54:29,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:54:29,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858469396, max timestamp=1681858469455Num uniques 859; 2023-04-18 22:54:29,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:54:29,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858469396, max timestamp=1681858469456Num uniques 859; 2023-04-18 22:54:29,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:54:29,476 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=535, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/c6e259f64b8e4e09931403312eee5a16 2023-04-18 22:54:29,479 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#C#compaction#176 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:29,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=468, cellSize=21428, totalHeapSize=48912, min timestamp=1681858469473, max timestamp=1681858469498Num uniques 1018; 2023-04-18 22:54:29,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=468, cellSize=21428, totalHeapSize=48912, min timestamp=1681858469473, max timestamp=1681858469498Num uniques 1018; 2023-04-18 22:54:29,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=21428, suffix heap size=133840,new segment heap size=48912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=468 2023-04-18 22:54:29,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=21428, suffix heap size=133840,new segment heap size=48912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=468 2023-04-18 22:54:29,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681858469478, max timestamp=1681858469498Num uniques 1018; 2023-04-18 22:54:29,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-18 22:54:29,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/f6603b7a207c4ede9824a79c4d4bc9b8 2023-04-18 22:54:29,535 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/5d41c8f2a63c477c8046830184406dc5 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/5d41c8f2a63c477c8046830184406dc5 2023-04-18 22:54:29,543 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:29,543 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/C of 052b0c71a65b8786b7851f4ab1dab728 into 5d41c8f2a63c477c8046830184406dc5(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:29,543 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:29,543 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/C, priority=8, startTime=1681858469354; duration=0sec 2023-04-18 22:54:29,543 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:29,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:29,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:29,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:29,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:29,546 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:29,546 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:29,546 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:29,546 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:29,547 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,547 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:29,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858529541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:29,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858529542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,549 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:29,549 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:29,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:29,550 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,550 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:29,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:29,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858529543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:29,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:29,558 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:29,558 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:29,558 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:29,559 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:29,559 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:29,559 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:29,559 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:29,559 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:29,560 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:29,560 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:29,560 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:29,560 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,560 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:29,560 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:29,562 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,562 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,562 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,562 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:29,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18d3b7a5[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:29,565 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:29,565 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,567 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,567 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:29,567 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:29,567 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:29,567 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,567 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:29,569 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:29,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:29,573 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:29,573 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:29,573 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:29,573 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:29,573 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:29,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,578 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,579 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,582 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,582 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,582 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,582 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:29,582 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:29,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:29,582 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:29,582 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:29,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:29,582 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,582 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,582 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,582 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:29,583 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:29,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:29,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:29,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858529649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:29,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858529650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:29,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858529653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,683 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:29,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:29,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:29,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:29,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:29,686 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:29,686 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:29,686 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:29,686 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:29,687 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,687 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,688 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:29,688 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:29,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:29,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,689 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,689 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:29,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:29,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:29,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:29,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:29,690 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:29,690 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:29,690 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:29,690 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:29,690 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:29,690 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:29,690 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:29,690 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:29,690 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:29,690 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:29,690 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:29,690 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,691 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:29,691 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:29,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:29,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@756c8140[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:29,694 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:29,694 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,697 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:29,697 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:29,697 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:29,697 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,697 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:29,699 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:29,699 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:29,699 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,699 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:29,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:29,701 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:29,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:29,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:29,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:29,701 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:29,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:29,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:29,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:29,704 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,705 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,709 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,709 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,709 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,709 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,709 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:29,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:29,709 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:29,709 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:29,709 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:29,709 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,710 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:29,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:29,710 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:29,710 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:29,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:29,810 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:29,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:29,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:29,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:29,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:29,813 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:29,813 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:29,813 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:29,813 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:29,814 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,814 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,816 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:29,816 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:29,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:29,816 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,817 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:29,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:29,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:29,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:29,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:29,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:29,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:29,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:29,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:29,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:29,818 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:29,818 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:29,818 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:29,818 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:29,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:29,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:29,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:29,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:29,820 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,820 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:29,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ddd4aba[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:29,822 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:29,822 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,823 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,824 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:29,824 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:29,824 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:29,824 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,824 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:29,833 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:29,833 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,833 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,833 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:29,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:29,837 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:29,837 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:29,837 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:29,837 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:29,837 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:29,839 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,839 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:29,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,839 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:29,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,842 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,844 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:29,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:29,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:29,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:29,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:29,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:29,846 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:29,847 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:29,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:29,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:29,847 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:29,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:29,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858529854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:29,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858529855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:29,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858529856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,947 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:29,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:29,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:29,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:29,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:29,951 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:29,951 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:29,951 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:29,951 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:29,951 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,951 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,952 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,952 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:29,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:29,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:29,953 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,953 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:29,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:29,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:29,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:29,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:29,954 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:29,954 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:29,954 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:29,954 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:29,954 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:29,954 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:29,954 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:29,954 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:29,954 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:29,954 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:29,955 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:29,955 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,955 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:29,955 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:29,957 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,957 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,957 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,958 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:29,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ec3cb3b[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:29,959 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:29,959 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,960 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,960 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:29,960 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:29,960 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,960 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:29,960 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:29,962 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,962 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:29,962 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/c30a8162081640d4acac115a00a76ea0 2023-04-18 22:54:29,962 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,962 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:29,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:29,967 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:29,967 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:29,967 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:29,967 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:29,967 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:29,968 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/c6e259f64b8e4e09931403312eee5a16 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/c6e259f64b8e4e09931403312eee5a16 2023-04-18 22:54:29,969 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,969 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:29,969 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:29,970 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,974 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,976 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,976 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,976 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:29,976 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:29,976 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,976 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:29,976 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:29,977 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:29,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:29,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:29,977 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:29,977 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:29,977 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/c6e259f64b8e4e09931403312eee5a16, entries=250, sequenceid=535, filesize=39.2 K 2023-04-18 22:54:29,977 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:29,977 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:29,978 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/f6603b7a207c4ede9824a79c4d4bc9b8 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/f6603b7a207c4ede9824a79c4d4bc9b8 2023-04-18 22:54:29,982 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/f6603b7a207c4ede9824a79c4d4bc9b8, entries=150, sequenceid=535, filesize=11.9 K 2023-04-18 22:54:29,983 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/c30a8162081640d4acac115a00a76ea0 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c30a8162081640d4acac115a00a76ea0 2023-04-18 22:54:29,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:29,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:29,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:29,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:29,987 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c30a8162081640d4acac115a00a76ea0, entries=150, sequenceid=535, filesize=11.9 K 2023-04-18 22:54:29,988 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~84.05 KB/86064, currentSize=158.78 KB/162590 for 052b0c71a65b8786b7851f4ab1dab728 in 603ms, sequenceid=535, compaction requested=false 2023-04-18 22:54:29,988 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:30,077 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:30,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:30,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:30,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:30,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:30,082 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:30,082 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:30,082 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:30,082 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:30,083 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:30,083 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:30,084 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,084 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,084 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:30,087 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:30,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:30,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:30,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:30,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:30,089 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:30,089 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:30,089 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:30,089 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:30,089 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:30,089 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:30,089 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:30,089 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:30,090 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:30,090 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:30,090 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=158.78 KB heapSize=435.31 KB 2023-04-18 22:54:30,090 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:30,090 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:30,090 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:30,090 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:30,090 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:30,090 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:30,110 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304183f5322708b684f19985815479f20bd2e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304183f5322708b684f19985815479f20bd2e 2023-04-18 22:54:30,118 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=590, memsize=55.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/289036a9cefd4f3094a1417a3caec241 2023-04-18 22:54:30,134 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/b560667f507f4cd1b7e84cbb4db294cc 2023-04-18 22:54:30,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:30,162 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:30,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:30,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858530166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:30,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:30,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858530169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858530169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:30,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858530237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:30,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858530237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:30,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858530270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:30,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858530277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:30,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858530277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:30,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858530473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:30,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858530479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:30,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858530482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,559 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/0a95e4d537434af59ccdbf6cc5bda17d 2023-04-18 22:54:30,567 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/289036a9cefd4f3094a1417a3caec241 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/289036a9cefd4f3094a1417a3caec241 2023-04-18 22:54:30,572 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/289036a9cefd4f3094a1417a3caec241, entries=150, sequenceid=590, filesize=25.4 K 2023-04-18 22:54:30,573 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/b560667f507f4cd1b7e84cbb4db294cc as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b560667f507f4cd1b7e84cbb4db294cc 2023-04-18 22:54:30,578 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b560667f507f4cd1b7e84cbb4db294cc, entries=150, sequenceid=590, filesize=11.9 K 2023-04-18 22:54:30,579 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/0a95e4d537434af59ccdbf6cc5bda17d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/0a95e4d537434af59ccdbf6cc5bda17d 2023-04-18 22:54:30,583 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/0a95e4d537434af59ccdbf6cc5bda17d, entries=150, sequenceid=590, filesize=11.9 K 2023-04-18 22:54:30,585 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~158.78 KB/162590, heapSize ~435.27 KB/445712, currentSize=33.54 KB/34350 for 052b0c71a65b8786b7851f4ab1dab728 in 495ms, sequenceid=590, compaction requested=true 2023-04-18 22:54:30,585 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:30,585 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:30,585 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:30,585 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:30,585 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,586 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:30,586 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:30,589 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:30,589 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:30,589 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:30,589 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:30,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:30,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ba83fd9[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:30,591 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:30,591 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:30,593 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:30,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:30,593 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:30,593 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:30,593 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:30,593 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,593 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:30,595 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,595 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:30,595 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:30,596 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:30,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:30,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:30,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:30,598 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:30,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:30,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:30,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:30,599 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:30,601 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:30,601 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:30,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:30,601 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:30,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:30,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:30,602 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:30,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:30,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:30,603 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,605 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,605 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:30,605 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:30,606 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:30,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:30,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:30,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:30,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:30,606 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:30,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:30,606 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:30,606 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,606 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:30,606 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:30,606 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:30,606 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:30,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:30,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:30,606 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:30,655 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/1fa647f0f4624837950e1ac269bdb4ca as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/1fa647f0f4624837950e1ac269bdb4ca 2023-04-18 22:54:30,662 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/A of 052b0c71a65b8786b7851f4ab1dab728 into 1fa647f0f4624837950e1ac269bdb4ca(size=25.9 K), total size for store is 90.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-18 22:54:30,662 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:30,662 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/A, priority=8, startTime=1681858469353; duration=1sec 2023-04-18 22:54:30,662 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:30,707 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:30,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:30,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:30,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:30,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:30,711 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:30,711 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:30,712 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:30,712 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:30,712 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:30,712 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:30,714 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:30,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:30,714 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:30,715 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,715 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:30,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:30,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:30,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:30,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:30,724 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:30,724 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:30,724 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:30,740 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:30,740 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:30,740 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:30,740 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:30,740 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-18 22:54:30,740 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:30,741 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:30,741 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:30,741 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:30,741 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:30,741 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:30,782 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418393a7e820c8f48bd9d29b69968dd36a5 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418393a7e820c8f48bd9d29b69968dd36a5 2023-04-18 22:54:30,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:30,798 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:30,802 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=599, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/98a7e710a80b42df8dd1d180d478da42 2023-04-18 22:54:30,819 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/1bc5072aa53a4fad8ac6b6ccfe4c15ee 2023-04-18 22:54:30,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681858470829, max timestamp=1681858470838Num uniques 859; 2023-04-18 22:54:30,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681858470829, max timestamp=1681858470838Num uniques 859; 2023-04-18 22:54:30,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681858470818, max timestamp=1681858470838Num uniques 859; 2023-04-18 22:54:30,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:30,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:54:30,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:30,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:30,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858530886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9970, totalHeapSize=22912, min timestamp=1681858470878, max timestamp=1681858470885Num uniques 1118; 2023-04-18 22:54:30,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51190, new segment data size=9970, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-18 22:54:30,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681858470878, max timestamp=1681858470885Num uniques 1068; 2023-04-18 22:54:30,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681858470878, max timestamp=1681858470885Num uniques 1068; 2023-04-18 22:54:30,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-18 22:54:30,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-18 22:54:30,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=427, cellSize=19552, totalHeapSize=44648, min timestamp=1681858470896, max timestamp=1681858470914Num uniques 1077; 2023-04-18 22:54:30,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=427, cellSize=19552, totalHeapSize=44648, min timestamp=1681858470896, max timestamp=1681858470914Num uniques 1077; 2023-04-18 22:54:30,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=19552, suffix heap size=139976,new segment heap size=44648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=427 2023-04-18 22:54:30,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=427, cellSize=19552, totalHeapSize=44648, min timestamp=1681858470896, max timestamp=1681858470914Num uniques 1077; 2023-04-18 22:54:30,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=19552, suffix heap size=139976,new segment heap size=44648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=427 2023-04-18 22:54:30,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=19552, suffix heap size=139976,new segment heap size=44648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=427 2023-04-18 22:54:30,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:30,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858530946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:30,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858530949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:30,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:30,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858530988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:31,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:31,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858531052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858531052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:31,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858531192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,234 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/e75fd267e1794489aae9150761e64c12 2023-04-18 22:54:31,238 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/98a7e710a80b42df8dd1d180d478da42 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/98a7e710a80b42df8dd1d180d478da42 2023-04-18 22:54:31,242 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/98a7e710a80b42df8dd1d180d478da42, entries=100, sequenceid=599, filesize=18.6 K 2023-04-18 22:54:31,243 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/1bc5072aa53a4fad8ac6b6ccfe4c15ee as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/1bc5072aa53a4fad8ac6b6ccfe4c15ee 2023-04-18 22:54:31,249 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/1bc5072aa53a4fad8ac6b6ccfe4c15ee, entries=100, sequenceid=599, filesize=9.5 K 2023-04-18 22:54:31,250 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/e75fd267e1794489aae9150761e64c12 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e75fd267e1794489aae9150761e64c12 2023-04-18 22:54:31,257 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e75fd267e1794489aae9150761e64c12, entries=100, sequenceid=599, filesize=9.5 K 2023-04-18 22:54:31,257 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for 052b0c71a65b8786b7851f4ab1dab728 in 517ms, sequenceid=599, compaction requested=true 2023-04-18 22:54:31,258 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:31,258 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:31,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:31,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:31,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:31,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:31,261 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,261 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,261 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,261 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:31,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:31,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a5fdd6d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:31,264 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:31,264 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:31,265 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=167.72 KB heapSize=459.47 KB 2023-04-18 22:54:31,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:31,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:31,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:31,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:31,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:31,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:31,266 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:31,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:31,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:31,266 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,266 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:31,268 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:31,268 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:31,268 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:31,268 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:31,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:31,271 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:31,272 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:31,272 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:31,272 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:31,272 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:31,275 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,275 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,275 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:31,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,277 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,279 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:31,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58972 deadline: 1681858531278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:31,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858531279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,287 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,287 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:31,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:31,287 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,287 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:31,287 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:31,287 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:31,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:31,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:31,288 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:31,288 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:31,287 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,288 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,288 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,288 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:31,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:31,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:31,324 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304185de41706f24e4e1982513f6d399a38e4 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304185de41706f24e4e1982513f6d399a38e4 2023-04-18 22:54:31,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:31,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858531383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:31,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58974 deadline: 1681858531383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,388 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:31,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:31,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:31,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:31,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:31,391 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:31,391 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:31,391 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:31,391 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:31,392 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,392 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,395 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:31,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:31,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:31,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:31,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:31,396 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:31,396 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:31,396 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:31,396 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:31,396 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:31,397 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:31,397 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:31,397 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:31,397 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:31,397 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:31,397 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:31,397 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,397 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:31,397 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:31,400 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,400 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,401 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,401 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:31,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d40d806[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:31,403 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:31,403 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,406 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,406 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:31,406 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:31,406 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:31,406 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,406 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:31,409 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:31,409 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,409 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,410 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:31,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:31,412 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:31,412 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:31,412 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:31,412 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:31,412 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:31,416 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,416 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,416 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:31,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:31,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:31,417 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,418 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,423 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,423 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,423 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:31,423 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:31,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:31,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:31,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:31,423 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,423 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,423 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,423 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,423 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:31,423 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:31,423 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:31,424 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:31,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:31,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:31,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:31,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858531495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,524 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:31,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:31,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:31,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:31,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:31,527 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:31,527 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:31,527 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:31,527 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:31,528 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,528 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,532 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,532 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:31,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:31,532 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,532 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:31,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:31,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:31,533 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:31,534 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:31,534 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:31,534 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:31,534 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:31,534 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:31,534 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:31,534 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:31,534 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:31,534 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:31,534 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:31,534 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,534 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:31,534 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:31,539 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,539 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,540 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,540 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:31,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dea5486[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:31,543 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:31,543 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,548 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,549 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:31,549 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,549 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:31,549 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:31,549 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:31,553 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,553 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:31,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:31,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:31,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:31,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:31,556 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:31,556 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:31,556 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:31,556 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:31,556 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:31,562 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,562 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,562 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,562 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:31,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,564 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,569 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:31,569 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:31,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:31,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:31,569 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:31,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:31,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:31,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:31,569 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,570 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,570 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,570 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,570 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,570 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:31,570 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:31,570 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,570 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:31,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:31,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:31,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858531589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:31,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858531589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,670 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:31,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:31,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:31,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:31,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:31,674 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:31,674 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:31,674 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:31,674 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:31,674 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,674 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,676 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,676 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:31,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:31,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:31,676 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,676 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:31,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:31,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:31,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:31,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:31,678 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:31,678 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:31,678 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:31,678 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:31,678 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:31,678 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:31,678 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:31,678 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:31,678 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:31,678 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:31,678 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:31,678 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,678 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:31,678 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:31,681 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,681 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,681 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,681 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:31,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:31,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77d6f615[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:31,683 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:31,683 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,684 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,685 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:31,685 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:31,685 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:31,685 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,685 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:31,688 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,688 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:31,688 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:31,688 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:31,690 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:31,691 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:31,691 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:31,691 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:31,691 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:31,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:31,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,693 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,694 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,702 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,702 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:31,703 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:31,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:31,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:31,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:31,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:31,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:31,702 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:31,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:31,703 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:31,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:31,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,703 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:31,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,731 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=670, memsize=55.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/4d93336e541d4f72acbf42f5d04c1e85 2023-04-18 22:54:31,746 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/ff5cf1b0c1fa4203950beaebccb29856 2023-04-18 22:54:31,803 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:31,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:31,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:31,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:31,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:31,806 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:31,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:31,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:31,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:31,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,809 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:31,809 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:31,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:31,809 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,809 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:31,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:31,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:31,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:31,810 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:31,810 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:31,810 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:31,814 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:31,814 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:31,814 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:31,814 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:31,814 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:31,814 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:31,814 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:31,814 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:31,815 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,815 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:31,815 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:31,816 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,816 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,817 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,817 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:31,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3326f937[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:31,818 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:31,818 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,820 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,820 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:31,820 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:31,820 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:31,820 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,820 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:31,822 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:31,822 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:31,822 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,822 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:31,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:31,824 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:31,824 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:31,824 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:31,824 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:31,824 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:31,827 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,827 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:31,827 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:31,828 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,830 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,832 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:31,832 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:31,832 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:31,832 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:31,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:31,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:31,832 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,833 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,833 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,833 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:31,833 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:31,833 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:31,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:31,833 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,833 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:31,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858531892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:31,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858531893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,933 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:31,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:31,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:31,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:31,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:31,936 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:31,936 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:31,936 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:31,936 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:31,937 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,937 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,938 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,938 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:31,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:31,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:31,938 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,938 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:31,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:31,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:31,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:31,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:31,940 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:31,940 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:31,940 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:31,940 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:31,942 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,942 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,942 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,942 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:31,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@188d97bd[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:31,944 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:31,944 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,946 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:31,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:31,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:31,946 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,946 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:31,947 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,947 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,947 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:31,947 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:31,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:31,949 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:31,949 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:31,949 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:31,949 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:31,949 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:31,951 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,951 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,951 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:31,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:31,951 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,953 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:31,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,956 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,956 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,956 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:31,956 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:31,956 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,956 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:31,956 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:31,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:31,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:31,956 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:31,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:31,956 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:31,956 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:31,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:31,956 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:31,957 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:31,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:32,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:32,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858532004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,057 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:32,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:32,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:32,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:32,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:32,067 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:32,067 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:32,068 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:32,068 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:32,068 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:32,068 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:32,070 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,070 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:32,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:32,071 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,071 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:32,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:32,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:32,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:32,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:32,079 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:32,079 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:32,079 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:32,079 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:32,079 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:32,079 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:32,079 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:32,080 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:32,080 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:32,080 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:32,080 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:32,080 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,080 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:32,080 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:32,082 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,082 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:32,082 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:32,082 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:32,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:32,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@578cbe4e[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:32,083 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:32,084 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:32,085 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:32,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:32,085 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:32,085 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:32,085 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:32,085 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,085 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:32,086 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,087 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:32,087 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,087 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:32,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:32,089 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:32,089 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:32,089 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:32,089 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:32,089 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:32,092 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:32,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:32,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:32,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:32,093 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:32,093 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:32,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:32,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:32,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:32,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:32,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:32,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:32,095 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:32,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,096 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:32,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:32,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,099 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:32,099 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:32,099 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,099 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:32,099 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:32,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:32,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:32,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:32,099 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:32,100 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:32,100 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:32,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:32,100 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,100 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:32,100 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:32,100 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:32,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:32,185 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/e9aed972cd134fcb8540d1758ea4dfd8 2023-04-18 22:54:32,190 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/4d93336e541d4f72acbf42f5d04c1e85 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/4d93336e541d4f72acbf42f5d04c1e85 2023-04-18 22:54:32,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/4d93336e541d4f72acbf42f5d04c1e85, entries=150, sequenceid=670, filesize=25.4 K 2023-04-18 22:54:32,195 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/ff5cf1b0c1fa4203950beaebccb29856 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/ff5cf1b0c1fa4203950beaebccb29856 2023-04-18 22:54:32,200 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:32,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:32,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:32,203 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/ff5cf1b0c1fa4203950beaebccb29856, entries=150, sequenceid=670, filesize=11.9 K 2023-04-18 22:54:32,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:32,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:32,204 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:32,204 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:32,204 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:32,204 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:32,204 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:32,204 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:32,205 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/e9aed972cd134fcb8540d1758ea4dfd8 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e9aed972cd134fcb8540d1758ea4dfd8 2023-04-18 22:54:32,209 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e9aed972cd134fcb8540d1758ea4dfd8, entries=150, sequenceid=670, filesize=11.9 K 2023-04-18 22:54:32,210 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~459.42 KB/470448, currentSize=26.84 KB/27480 for 052b0c71a65b8786b7851f4ab1dab728 in 946ms, sequenceid=670, compaction requested=true 2023-04-18 22:54:32,210 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:32,210 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:32,211 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 22:54:32,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:32,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-18 22:54:32,211 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 22:54:32,211 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:32,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:32,211 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:32,212 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,212 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:32,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:32,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:32,214 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58998 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 22:54:32,214 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/B is initiating minor compaction (all files) 2023-04-18 22:54:32,214 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/B in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:32,214 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137778 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 22:54:32,214 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/24c3af227571410cb0838dfded867043, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/f6603b7a207c4ede9824a79c4d4bc9b8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b560667f507f4cd1b7e84cbb4db294cc, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/1bc5072aa53a4fad8ac6b6ccfe4c15ee, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/ff5cf1b0c1fa4203950beaebccb29856] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=57.6 K 2023-04-18 22:54:32,214 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/A is initiating minor compaction (all files) 2023-04-18 22:54:32,214 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/A in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:32,215 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/1fa647f0f4624837950e1ac269bdb4ca, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/c6e259f64b8e4e09931403312eee5a16, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/289036a9cefd4f3094a1417a3caec241, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/98a7e710a80b42df8dd1d180d478da42, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/4d93336e541d4f72acbf42f5d04c1e85] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=134.5 K 2023-04-18 22:54:32,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:32,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:32,215 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:32,215 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:32,215 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:32,216 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 1fa647f0f4624837950e1ac269bdb4ca, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1681858467055 2023-04-18 22:54:32,216 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 24c3af227571410cb0838dfded867043, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1681858467055 2023-04-18 22:54:32,216 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting c6e259f64b8e4e09931403312eee5a16, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681858468083 2023-04-18 22:54:32,216 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting f6603b7a207c4ede9824a79c4d4bc9b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681858468083 2023-04-18 22:54:32,217 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting b560667f507f4cd1b7e84cbb4db294cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681858469539 2023-04-18 22:54:32,217 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 289036a9cefd4f3094a1417a3caec241, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681858469539 2023-04-18 22:54:32,218 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 1bc5072aa53a4fad8ac6b6ccfe4c15ee, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1681858470162 2023-04-18 22:54:32,218 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 98a7e710a80b42df8dd1d180d478da42, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1681858470162 2023-04-18 22:54:32,218 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting ff5cf1b0c1fa4203950beaebccb29856, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1681858470936 2023-04-18 22:54:32,218 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 4d93336e541d4f72acbf42f5d04c1e85, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1681858470936 2023-04-18 22:54:32,220 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:32,220 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:32,220 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:32,220 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:32,221 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-18 22:54:32,221 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:32,221 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:32,221 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:32,221 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:32,222 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:32,222 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:32,237 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#B#compaction#188 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:32,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:32,253 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:32,294 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#A#compaction#189 average throughput is 0.51 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:32,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681858472284, max timestamp=1681858472294Num uniques 859; 2023-04-18 22:54:32,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:32,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681858472284, max timestamp=1681858472294Num uniques 859; 2023-04-18 22:54:32,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858472284, max timestamp=1681858472294Num uniques 859; 2023-04-18 22:54:32,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:32,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:32,309 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/4ab1fb5e4657422ca355388d7cf76169 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/4ab1fb5e4657422ca355388d7cf76169 2023-04-18 22:54:32,315 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304189a43571c215b42dc8f997ddf93632712 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304189a43571c215b42dc8f997ddf93632712 2023-04-18 22:54:32,316 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/B of 052b0c71a65b8786b7851f4ab1dab728 into 4ab1fb5e4657422ca355388d7cf76169(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:32,316 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:32,316 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/B, priority=11, startTime=1681858472211; duration=0sec 2023-04-18 22:54:32,316 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:32,317 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 22:54:32,330 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58998 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 22:54:32,330 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/C is initiating minor compaction (all files) 2023-04-18 22:54:32,330 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/C in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:32,330 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/5d41c8f2a63c477c8046830184406dc5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c30a8162081640d4acac115a00a76ea0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/0a95e4d537434af59ccdbf6cc5bda17d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e75fd267e1794489aae9150761e64c12, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e9aed972cd134fcb8540d1758ea4dfd8] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=57.6 K 2023-04-18 22:54:32,331 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 5d41c8f2a63c477c8046830184406dc5, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1681858467055 2023-04-18 22:54:32,332 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting c30a8162081640d4acac115a00a76ea0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681858468083 2023-04-18 22:54:32,333 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 0a95e4d537434af59ccdbf6cc5bda17d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681858469539 2023-04-18 22:54:32,333 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting e75fd267e1794489aae9150761e64c12, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1681858470162 2023-04-18 22:54:32,334 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting e9aed972cd134fcb8540d1758ea4dfd8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1681858470936 2023-04-18 22:54:32,346 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#C#compaction#191 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:32,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681858472333, max timestamp=1681858472352Num uniques 1068; 2023-04-18 22:54:32,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-18 22:54:32,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681858472333, max timestamp=1681858472352Num uniques 1068; 2023-04-18 22:54:32,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-18 22:54:32,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681858472333, max timestamp=1681858472352Num uniques 1068; 2023-04-18 22:54:32,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-18 22:54:32,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681858472352, max timestamp=1681858472390Num uniques 1077; 2023-04-18 22:54:32,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=14972, suffix heap size=139976,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=327 2023-04-18 22:54:32,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681858472352, max timestamp=1681858472390Num uniques 1077; 2023-04-18 22:54:32,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681858472352, max timestamp=1681858472390Num uniques 1077; 2023-04-18 22:54:32,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=14972, suffix heap size=139976,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=327 2023-04-18 22:54:32,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=14972, suffix heap size=139976,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=327 2023-04-18 22:54:32,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681858472412, max timestamp=1681858472419Num uniques 1186; 2023-04-18 22:54:32,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=10806, suffix heap size=151312,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=236 2023-04-18 22:54:32,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681858472412, max timestamp=1681858472419Num uniques 1186; 2023-04-18 22:54:32,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=10806, suffix heap size=151312,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=236 2023-04-18 22:54:32,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681858472412, max timestamp=1681858472419Num uniques 1186; 2023-04-18 22:54:32,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=10806, suffix heap size=151312,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=236 2023-04-18 22:54:32,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1681858472431, max timestamp=1681858472448Num uniques 1095; 2023-04-18 22:54:32,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=18090, suffix heap size=141848,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=395 2023-04-18 22:54:32,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1681858472431, max timestamp=1681858472448Num uniques 1095; 2023-04-18 22:54:32,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=18090, suffix heap size=141848,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=395 2023-04-18 22:54:32,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1681858472431, max timestamp=1681858472448Num uniques 1095; 2023-04-18 22:54:32,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=18090, suffix heap size=141848,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=395 2023-04-18 22:54:32,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:32,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858532497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:32,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858532497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:32,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858532497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:32,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858532498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:32,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858532604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:32,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858532604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:32,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858532604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:32,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858532604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,739 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=677, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/d56e9601ec1b437ca90fc773f358fe7d 2023-04-18 22:54:32,763 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/c2ba8162b5de49d8b21aad1998afcf02 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c2ba8162b5de49d8b21aad1998afcf02 2023-04-18 22:54:32,768 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/C of 052b0c71a65b8786b7851f4ab1dab728 into c2ba8162b5de49d8b21aad1998afcf02(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:32,768 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:32,768 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/C, priority=11, startTime=1681858472211; duration=0sec 2023-04-18 22:54:32,768 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:32,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:32,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858532808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:32,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858532808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:32,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858532808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:32,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:32,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858532809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:33,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59026 deadline: 1681858533011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:33,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858533110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:33,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858533110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:33,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858533112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:33,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858533115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,150 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/f66707c4967745d188b604a92d21fa1e 2023-04-18 22:54:33,548 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/bd7e82a5aff14f469d1aaace2878b586 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/bd7e82a5aff14f469d1aaace2878b586 2023-04-18 22:54:33,553 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/A of 052b0c71a65b8786b7851f4ab1dab728 into bd7e82a5aff14f469d1aaace2878b586(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-18 22:54:33,553 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:33,553 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/A, priority=11, startTime=1681858472210; duration=1sec 2023-04-18 22:54:33,553 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:33,573 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/0e221531b8044cafa1ebd57d2fbac449 2023-04-18 22:54:33,577 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/d56e9601ec1b437ca90fc773f358fe7d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/d56e9601ec1b437ca90fc773f358fe7d 2023-04-18 22:54:33,581 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/d56e9601ec1b437ca90fc773f358fe7d, entries=100, sequenceid=677, filesize=18.6 K 2023-04-18 22:54:33,582 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/f66707c4967745d188b604a92d21fa1e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/f66707c4967745d188b604a92d21fa1e 2023-04-18 22:54:33,586 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/f66707c4967745d188b604a92d21fa1e, entries=100, sequenceid=677, filesize=9.5 K 2023-04-18 22:54:33,587 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/0e221531b8044cafa1ebd57d2fbac449 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/0e221531b8044cafa1ebd57d2fbac449 2023-04-18 22:54:33,592 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/0e221531b8044cafa1ebd57d2fbac449, entries=100, sequenceid=677, filesize=9.5 K 2023-04-18 22:54:33,593 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 052b0c71a65b8786b7851f4ab1dab728 in 1372ms, sequenceid=677, compaction requested=false 2023-04-18 22:54:33,593 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:33,593 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:33,593 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:33,593 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:33,593 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,593 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:33,593 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:33,595 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,595 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:33,595 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,596 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:33,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:33,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@172a8cf4[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:33,597 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:33,597 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,600 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,600 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:33,600 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:33,600 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:33,600 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,600 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:33,601 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:33,601 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,601 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:33,602 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:33,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:33,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:33,604 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:33,604 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:33,605 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:33,605 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:33,605 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:33,606 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,606 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:33,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:33,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,606 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:33,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:33,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:33,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:33,608 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:33,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,610 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,612 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,612 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:33,612 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:33,613 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:33,613 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:33,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:33,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:33,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:33,613 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:33,613 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,613 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:33,613 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,613 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:33,613 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,613 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:33,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:33,617 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=167.72 KB heapSize=462.70 KB 2023-04-18 22:54:33,617 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:33,617 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:54:33,617 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:33,617 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:54:33,617 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:33,617 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:54:33,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:33,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858533619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:33,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858533621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:33,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858533622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:33,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858533623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,713 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:33,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:33,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:33,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:33,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:33,717 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:33,717 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:33,717 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:33,717 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:33,717 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,717 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:33,720 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:33,720 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:33,720 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,720 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:33,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:33,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:33,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:33,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:33,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:33,721 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:33,721 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:33,721 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:33,722 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:33,722 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:33,722 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:33,722 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:33,722 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:33,722 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:33,722 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:33,722 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:33,722 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,722 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:33,722 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:33,724 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,724 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:33,724 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,724 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:33,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13c13067[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:33,726 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:33,726 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,728 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,728 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:33,728 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:33,728 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:33,728 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,728 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:33,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:33,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858533726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,729 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:33,729 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:33,729 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:33,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858533726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,730 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:33,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:33,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858533727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:33,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:33,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:33,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:33,732 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:33,733 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:33,733 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:33,733 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:33,733 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:33,735 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,735 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:33,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:33,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,735 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:33,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:33,736 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:33,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,737 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:33,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,739 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,739 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:33,739 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:33,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:33,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:33,739 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:33,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:33,739 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:33,739 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:33,739 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:33,739 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:33,740 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:33,740 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:33,740 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:33,740 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:33,840 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:33,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:33,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:33,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:33,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:33,843 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:33,843 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:33,843 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:33,843 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:33,844 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,844 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:33,845 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:33,845 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:33,846 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,846 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:33,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:33,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:33,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:33,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:33,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:33,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:33,847 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:33,847 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:33,847 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:33,847 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:33,847 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:33,847 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:33,847 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:33,847 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:33,848 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:33,848 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:33,848 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:33,848 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,848 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:33,848 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:33,852 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,852 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:33,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:33,852 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,852 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:33,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:33,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67856ddd[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:33,854 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:33,854 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,856 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,856 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:33,856 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:33,856 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:33,856 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,856 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:33,858 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:33,858 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:33,858 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,858 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:33,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:33,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:33,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:33,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:33,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:33,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:33,861 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:33,862 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:33,862 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:33,862 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:33,862 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:33,863 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,863 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:33,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,863 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:33,864 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:33,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,865 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:33,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:33,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:33,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:33,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,868 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:33,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:33,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:33,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:33,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:33,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:33,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:33,868 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:33,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:33,869 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:33,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:33,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:33,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:33,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858533931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:33,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858533931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:33,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858533932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,969 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:33,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:33,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:33,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:33,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:33,972 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:33,972 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:33,972 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:33,972 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:33,973 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,973 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:33,975 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:33,975 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:33,975 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,975 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:33,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:33,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:33,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:33,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:33,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:33,977 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:33,977 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:33,977 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:33,978 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:33,978 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:33,978 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:33,978 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:33,978 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:33,978 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:33,978 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:33,978 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:33,978 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,978 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:33,978 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:33,979 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,979 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:33,980 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,980 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:33,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:33,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55c9c1ad[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:33,981 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:33,981 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,986 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,986 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:33,986 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:33,986 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:33,987 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,987 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:33,989 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:33,989 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:33,989 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:33,989 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:33,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:33,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:33,991 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:33,991 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:33,991 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:33,991 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:33,991 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:33,993 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,993 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,993 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:33,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:33,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:33,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:33,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:33,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:33,994 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:33,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,995 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:33,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:33,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,997 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,997 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:33,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:33,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:33,997 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:33,997 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:33,998 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:33,998 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:33,998 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:33,998 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:33,997 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:33,998 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,998 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:33,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:33,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:33,998 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:33,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:33,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:33,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:34,032 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304183b20ea261881478499787c947f70cd44 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304183b20ea261881478499787c947f70cd44 2023-04-18 22:54:34,038 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=787, memsize=58.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/77f88f59f7c7425eb6b8688445c8b9e7 2023-04-18 22:54:34,098 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:34,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:34,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:34,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:34,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:34,101 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:34,102 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:34,102 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:34,102 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:34,102 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,102 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,104 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:34,104 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:34,104 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,104 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:34,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:34,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:34,105 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:34,105 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:34,105 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:34,105 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:34,105 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:34,105 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:34,106 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:34,106 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:34,106 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:34,106 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:34,106 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:34,106 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,106 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:34,106 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:34,109 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,109 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,109 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,109 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:34,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1647bacc[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:34,110 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:34,111 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,111 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,112 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:34,112 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:34,112 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:34,112 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,112 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:34,113 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:34,113 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,113 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,113 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:34,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:34,116 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:34,116 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:34,116 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:34,116 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:34,116 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:34,118 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,118 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,118 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:34,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:34,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:34,119 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,120 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,125 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,126 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,126 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,126 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:34,126 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:34,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:34,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:34,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:34,126 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:34,126 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,126 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,126 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,126 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:34,126 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:34,126 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:34,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:34,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:34,226 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:34,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:34,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:34,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:34,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:34,229 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:34,229 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:34,229 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:34,229 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:34,230 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,230 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,231 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:34,231 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:34,231 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,231 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:34,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:34,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:34,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:34,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:34,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:34,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:34,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:34,233 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:34,233 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:34,233 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:34,233 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:34,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:34,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:34,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:34,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:34,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:34,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858534234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,235 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,235 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:34,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858534235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:34,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858534236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,236 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,236 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:34,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c9ff26b[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:34,237 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:34,237 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,239 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,239 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:34,239 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:34,239 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,239 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:34,239 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:34,241 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:34,241 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:34,241 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,241 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:34,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:34,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:34,243 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:34,243 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:34,243 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:34,244 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:34,244 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:34,245 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,245 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,245 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:34,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:34,246 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,247 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,249 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:34,249 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:34,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:34,249 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:34,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:34,249 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:34,249 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:34,250 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,250 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:34,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:34,250 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:34,250 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,250 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,250 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,250 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,350 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:34,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:34,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:34,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:34,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:34,353 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:34,353 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:34,354 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:34,354 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:34,354 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,354 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,355 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:34,355 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:34,356 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,356 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:34,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:34,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:34,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:34,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:34,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:34,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:34,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:34,357 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:34,358 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:34,358 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:34,358 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:34,358 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:34,358 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:34,358 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,358 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:34,358 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:34,361 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,361 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,361 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,361 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:34,362 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:34,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40b88da0[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:34,363 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,364 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,364 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:34,364 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:34,364 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:34,364 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,364 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:34,368 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:34,368 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:34,368 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,368 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:34,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:34,370 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:34,370 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:34,370 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:34,370 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:34,370 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:34,372 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,372 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,372 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:34,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,373 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,374 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,376 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:34,376 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:34,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:34,376 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:34,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:34,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:34,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:34,376 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:34,376 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:34,376 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,376 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:34,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:34,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:34,377 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,377 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,377 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,377 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,377 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,450 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=787 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/97bbe2d797704ac481e4f3b3bc9e7504 2023-04-18 22:54:34,477 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:34,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:34,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:34,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:34,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:34,481 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:34,481 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:34,481 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:34,481 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:34,481 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,482 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,484 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:34,484 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:34,484 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,484 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:34,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:34,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:34,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:34,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:34,486 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:34,489 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:34,489 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:34,489 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:34,490 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:34,490 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:34,490 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:34,490 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:34,490 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:34,490 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,490 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:34,490 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:34,492 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,492 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,492 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,492 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:34,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5be7e8b3[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:34,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:34,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,496 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:34,496 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:34,496 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:34,496 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,496 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:34,498 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:34,498 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:34,498 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:34,498 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:34,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:34,501 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:34,501 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:34,501 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:34,501 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:34,501 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:34,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:34,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:34,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:34,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,507 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,510 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,510 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:34,510 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:34,510 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,510 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:34,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:34,510 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,510 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:34,510 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:34,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:34,510 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,511 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,511 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:34,511 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:34,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:34,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:34,611 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:34,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:34,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:34,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:34,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:34,615 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:34,615 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:34,615 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:34,615 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:34,616 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,616 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,617 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:34,617 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:34,617 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,617 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:34,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:34,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:34,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:34,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:34,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:34,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:34,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:34,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:34,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:34,619 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:34,619 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:34,619 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:34,619 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:34,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:34,620 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:34,620 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,620 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:34,620 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:34,622 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,622 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,623 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,623 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:34,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@705db8d9[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:34,624 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:34,624 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,626 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,626 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:34,626 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:34,626 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:34,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:34,626 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,627 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:34,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858534625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,629 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,629 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:34,629 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,629 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:34,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:34,632 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:34,632 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:34,632 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:34,632 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:34,632 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:34,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:34,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,635 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,637 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,640 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,640 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,640 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:34,640 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:34,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:34,640 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:34,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:34,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:34,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:34,641 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:34,640 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:34,640 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,641 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,641 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,641 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:34,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:34,641 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:34,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858534737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:34,741 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:34,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858534739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:34,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:34,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858534740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:34,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:34,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:34,745 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:34,745 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:34,745 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:34,745 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:34,745 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,745 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,751 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,751 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:34,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:34,751 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,751 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:34,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:34,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:34,753 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:34,753 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:34,753 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:34,753 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:34,753 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:34,753 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:34,753 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:34,753 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:34,753 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:34,753 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:34,753 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:34,753 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,753 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:34,753 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:34,755 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,755 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,755 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,755 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:34,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3220013[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:34,757 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:34,757 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,758 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,758 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:34,759 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:34,759 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,759 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:34,759 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:34,760 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,760 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:34,760 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,760 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:34,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:34,762 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:34,762 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:34,762 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:34,762 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:34,763 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:34,765 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,765 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,765 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:34,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:34,766 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,767 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,772 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,772 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,772 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,772 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:34,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:34,772 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:34,772 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:34,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:34,772 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:34,772 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:34,773 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:34,772 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,772 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:34,773 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:34,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:34,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:34,862 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=787 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/3543ec11040f40558abe588a5658c415 2023-04-18 22:54:34,867 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/77f88f59f7c7425eb6b8688445c8b9e7 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/77f88f59f7c7425eb6b8688445c8b9e7 2023-04-18 22:54:34,872 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/77f88f59f7c7425eb6b8688445c8b9e7, entries=250, sequenceid=787, filesize=39.2 K 2023-04-18 22:54:34,872 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/97bbe2d797704ac481e4f3b3bc9e7504 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/97bbe2d797704ac481e4f3b3bc9e7504 2023-04-18 22:54:34,873 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:34,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:34,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:34,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:34,877 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/97bbe2d797704ac481e4f3b3bc9e7504, entries=150, sequenceid=787, filesize=11.9 K 2023-04-18 22:54:34,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:34,877 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:34,877 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:34,877 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:34,877 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:34,877 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,877 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,878 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/3543ec11040f40558abe588a5658c415 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/3543ec11040f40558abe588a5658c415 2023-04-18 22:54:34,879 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:34,879 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:34,879 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,880 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:34,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:34,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:34,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:34,881 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:34,881 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:34,881 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:34,881 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:34,882 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/3543ec11040f40558abe588a5658c415, entries=150, sequenceid=787, filesize=11.9 K 2023-04-18 22:54:34,883 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,883 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,884 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~174.43 KB/178620, heapSize ~483.05 KB/494640, currentSize=20.13 KB/20610 for 052b0c71a65b8786b7851f4ab1dab728 in 1266ms, sequenceid=787, compaction requested=true 2023-04-18 22:54:34,884 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:34,884 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:34,884 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:34,884 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:34,884 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:34,884 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:34,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,884 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,884 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:34,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,885 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,885 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/B is initiating minor compaction (all files) 2023-04-18 22:54:34,885 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85856 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:34,886 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/B in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:34,886 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/A is initiating minor compaction (all files) 2023-04-18 22:54:34,886 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/4ab1fb5e4657422ca355388d7cf76169, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/f66707c4967745d188b604a92d21fa1e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/97bbe2d797704ac481e4f3b3bc9e7504] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=34.0 K 2023-04-18 22:54:34,886 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/A in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:34,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,886 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/bd7e82a5aff14f469d1aaace2878b586, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/d56e9601ec1b437ca90fc773f358fe7d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/77f88f59f7c7425eb6b8688445c8b9e7] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=83.8 K 2023-04-18 22:54:34,886 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 4ab1fb5e4657422ca355388d7cf76169, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1681858470936 2023-04-18 22:54:34,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:34,886 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting bd7e82a5aff14f469d1aaace2878b586, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1681858470936 2023-04-18 22:54:34,886 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:34,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@773485a[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:34,886 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,887 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting f66707c4967745d188b604a92d21fa1e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1681858471268 2023-04-18 22:54:34,887 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting d56e9601ec1b437ca90fc773f358fe7d, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1681858471268 2023-04-18 22:54:34,887 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 97bbe2d797704ac481e4f3b3bc9e7504, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1681858472468 2023-04-18 22:54:34,887 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 77f88f59f7c7425eb6b8688445c8b9e7, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1681858472468 2023-04-18 22:54:34,892 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,892 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:34,892 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:34,892 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:34,892 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,892 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:34,894 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:34,894 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,894 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,894 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:34,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:34,897 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:34,897 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:34,897 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:34,898 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:34,898 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:34,899 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:34,900 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,900 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:34,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:34,901 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:34,903 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,905 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:34,906 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:34,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:34,906 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:34,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:34,906 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,906 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:34,906 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:34,906 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,906 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:34,906 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:34,906 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:34,906 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:34,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:34,906 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:34,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:34,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:34,907 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#B#compaction#198 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:34,937 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#A#compaction#197 average throughput is 0.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:34,966 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/d3f6541c5b9c415694b1ef773dba579c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/d3f6541c5b9c415694b1ef773dba579c 2023-04-18 22:54:34,971 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/B of 052b0c71a65b8786b7851f4ab1dab728 into d3f6541c5b9c415694b1ef773dba579c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:34,972 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:34,972 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/B, priority=13, startTime=1681858474884; duration=0sec 2023-04-18 22:54:34,972 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:34,972 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:34,973 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:34,973 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/C is initiating minor compaction (all files) 2023-04-18 22:54:34,973 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/C in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:34,973 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c2ba8162b5de49d8b21aad1998afcf02, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/0e221531b8044cafa1ebd57d2fbac449, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/3543ec11040f40558abe588a5658c415] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=34.0 K 2023-04-18 22:54:34,974 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting c2ba8162b5de49d8b21aad1998afcf02, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1681858470936 2023-04-18 22:54:34,974 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 0e221531b8044cafa1ebd57d2fbac449, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1681858471268 2023-04-18 22:54:34,974 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 3543ec11040f40558abe588a5658c415, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1681858472468 2023-04-18 22:54:34,984 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#C#compaction#199 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:34,994 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/2957c8377d6741a7afa092c9ed77cf98 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/2957c8377d6741a7afa092c9ed77cf98 2023-04-18 22:54:35,000 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/C of 052b0c71a65b8786b7851f4ab1dab728 into 2957c8377d6741a7afa092c9ed77cf98(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:35,000 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:35,000 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/C, priority=13, startTime=1681858474884; duration=0sec 2023-04-18 22:54:35,000 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:35,006 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:35,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:35,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:35,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:35,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:35,016 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:35,016 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:35,016 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:35,016 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:35,016 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:35,016 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:35,018 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:35,018 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:35,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:35,019 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:35,019 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:35,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:35,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:35,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:35,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:35,020 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:35,020 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:35,020 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:35,020 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:35,020 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:35,020 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:35,020 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:35,020 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-18 22:54:35,021 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:35,021 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:35,021 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:35,021 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:35,021 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:35,021 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:35,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:35,042 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:35,092 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304182c911c5660ab4a34a930f766c8ebeba8 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304182c911c5660ab4a34a930f766c8ebeba8 2023-04-18 22:54:35,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858475096, max timestamp=1681858475113Num uniques 859; 2023-04-18 22:54:35,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:35,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858475096, max timestamp=1681858475113Num uniques 859; 2023-04-18 22:54:35,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858475096, max timestamp=1681858475113Num uniques 859; 2023-04-18 22:54:35,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:35,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:35,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858475136, max timestamp=1681858475163Num uniques 1068; 2023-04-18 22:54:35,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858475136, max timestamp=1681858475163Num uniques 1068; 2023-04-18 22:54:35,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858475136, max timestamp=1681858475163Num uniques 1068; 2023-04-18 22:54:35,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-18 22:54:35,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-18 22:54:35,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-18 22:54:35,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681858475203, max timestamp=1681858475220Num uniques 1027; 2023-04-18 22:54:35,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681858475203, max timestamp=1681858475220Num uniques 1027; 2023-04-18 22:54:35,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 22:54:35,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681858475203, max timestamp=1681858475220Num uniques 1027; 2023-04-18 22:54:35,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 22:54:35,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 22:54:35,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681858475246, max timestamp=1681858475275Num uniques 1036; 2023-04-18 22:54:35,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681858475246, max timestamp=1681858475275Num uniques 1036; 2023-04-18 22:54:35,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 22:54:35,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681858475246, max timestamp=1681858475275Num uniques 1036; 2023-04-18 22:54:35,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 22:54:35,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 22:54:35,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681858475322, max timestamp=1681858475328Num uniques 1045; 2023-04-18 22:54:35,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681858475322, max timestamp=1681858475328Num uniques 1045; 2023-04-18 22:54:35,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-18 22:54:35,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-18 22:54:35,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681858475322, max timestamp=1681858475328Num uniques 1045; 2023-04-18 22:54:35,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-18 22:54:35,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681858475345, max timestamp=1681858475381Num uniques 1054; 2023-04-18 22:54:35,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-18 22:54:35,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681858475345, max timestamp=1681858475381Num uniques 1054; 2023-04-18 22:54:35,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681858475345, max timestamp=1681858475381Num uniques 1054; 2023-04-18 22:54:35,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-18 22:54:35,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-18 22:54:35,398 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/130e35ea0db64977b8644b85f3de4a4a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/130e35ea0db64977b8644b85f3de4a4a 2023-04-18 22:54:35,403 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/A of 052b0c71a65b8786b7851f4ab1dab728 into 130e35ea0db64977b8644b85f3de4a4a(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:35,403 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:35,403 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/A, priority=13, startTime=1681858474884; duration=0sec 2023-04-18 22:54:35,403 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:35,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681858475422, max timestamp=1681858475442Num uniques 1113; 2023-04-18 22:54:35,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-18 22:54:35,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681858475422, max timestamp=1681858475442Num uniques 1113; 2023-04-18 22:54:35,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681858475422, max timestamp=1681858475442Num uniques 1113; 2023-04-18 22:54:35,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-18 22:54:35,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-18 22:54:35,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681858475487, max timestamp=1681858475500Num uniques 1072; 2023-04-18 22:54:35,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681858475487, max timestamp=1681858475500Num uniques 1072; 2023-04-18 22:54:35,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681858475487, max timestamp=1681858475500Num uniques 1072; 2023-04-18 22:54:35,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-18 22:54:35,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-18 22:54:35,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-18 22:54:35,506 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=795, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/cfb8be4cbac74b03b45893465da198d3 2023-04-18 22:54:35,528 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/11c9909072f844d48e9059e07be395f7 2023-04-18 22:54:35,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681858475551, max timestamp=1681858475566Num uniques 1081; 2023-04-18 22:54:35,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681858475551, max timestamp=1681858475566Num uniques 1081; 2023-04-18 22:54:35,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-18 22:54:35,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681858475551, max timestamp=1681858475566Num uniques 1081; 2023-04-18 22:54:35,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-18 22:54:35,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-18 22:54:35,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681858475615, max timestamp=1681858475632Num uniques 1040; 2023-04-18 22:54:35,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-18 22:54:35,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681858475615, max timestamp=1681858475632Num uniques 1040; 2023-04-18 22:54:35,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681858475615, max timestamp=1681858475632Num uniques 1040; 2023-04-18 22:54:35,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-18 22:54:35,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-18 22:54:35,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681858475669, max timestamp=1681858475686Num uniques 1099; 2023-04-18 22:54:35,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-18 22:54:35,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681858475669, max timestamp=1681858475686Num uniques 1099; 2023-04-18 22:54:35,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-18 22:54:35,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681858475669, max timestamp=1681858475686Num uniques 1099; 2023-04-18 22:54:35,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-18 22:54:35,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681858475734, max timestamp=1681858475746Num uniques 1058; 2023-04-18 22:54:35,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-18 22:54:35,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681858475734, max timestamp=1681858475746Num uniques 1058; 2023-04-18 22:54:35,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681858475734, max timestamp=1681858475746Num uniques 1058; 2023-04-18 22:54:35,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-18 22:54:35,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-18 22:54:35,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681858475761, max timestamp=1681858475789Num uniques 1017; 2023-04-18 22:54:35,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=12222, suffix heap size=133736,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=267 2023-04-18 22:54:35,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681858475761, max timestamp=1681858475789Num uniques 1017; 2023-04-18 22:54:35,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=12222, suffix heap size=133736,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=267 2023-04-18 22:54:35,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681858475761, max timestamp=1681858475789Num uniques 1017; 2023-04-18 22:54:35,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=12222, suffix heap size=133736,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=267 2023-04-18 22:54:35,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681858475801, max timestamp=1681858475810Num uniques 1126; 2023-04-18 22:54:35,812 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-04-18 22:54:35,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681858475801, max timestamp=1681858475810Num uniques 1126; 2023-04-18 22:54:35,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-04-18 22:54:35,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681858475801, max timestamp=1681858475810Num uniques 1126; 2023-04-18 22:54:35,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-04-18 22:54:35,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8460, totalHeapSize=19480, min timestamp=1681858475819, max timestamp=1681858475860Num uniques 1085; 2023-04-18 22:54:35,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49680, new segment data size=8460, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-18 22:54:35,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681858475819, max timestamp=1681858475859Num uniques 1085; 2023-04-18 22:54:35,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8460, totalHeapSize=19480, min timestamp=1681858475819, max timestamp=1681858475860Num uniques 1085; 2023-04-18 22:54:35,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-18 22:54:35,862 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49680, new segment data size=8460, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-18 22:54:35,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681858475871, max timestamp=1681858475880Num uniques 1044; 2023-04-18 22:54:35,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-18 22:54:35,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681858475871, max timestamp=1681858475880Num uniques 1044; 2023-04-18 22:54:35,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-18 22:54:35,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681858475871, max timestamp=1681858475880Num uniques 1044; 2023-04-18 22:54:35,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-18 22:54:35,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681858475894, max timestamp=1681858475899Num uniques 1053; 2023-04-18 22:54:35,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-18 22:54:35,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681858475894, max timestamp=1681858475899Num uniques 1053; 2023-04-18 22:54:35,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-18 22:54:35,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681858475894, max timestamp=1681858475899Num uniques 1053; 2023-04-18 22:54:35,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-04-18 22:54:35,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1681858475909, max timestamp=1681858475917Num uniques 1012; 2023-04-18 22:54:35,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=14282, suffix heap size=133216,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=312 2023-04-18 22:54:35,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1681858475909, max timestamp=1681858475917Num uniques 1012; 2023-04-18 22:54:35,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=14282, suffix heap size=133216,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=312 2023-04-18 22:54:35,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=312, cellSize=14282, totalHeapSize=32688, min timestamp=1681858475909, max timestamp=1681858475917Num uniques 1012; 2023-04-18 22:54:35,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=14282, suffix heap size=133216,new segment heap size=32688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=312 2023-04-18 22:54:35,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681858475932, max timestamp=1681858475935Num uniques 1171; 2023-04-18 22:54:35,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681858475932, max timestamp=1681858475935Num uniques 1171; 2023-04-18 22:54:35,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681858475932, max timestamp=1681858475935Num uniques 1171; 2023-04-18 22:54:35,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=10116, suffix heap size=149752,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=221 2023-04-18 22:54:35,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=10116, suffix heap size=149752,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=221 2023-04-18 22:54:35,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53626, new segment data size=10116, suffix heap size=149752,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1171, new segment cells count=221 2023-04-18 22:54:35,945 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/bab32f3b9eb04193ba339dd92b2ec49c 2023-04-18 22:54:35,950 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/cfb8be4cbac74b03b45893465da198d3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/cfb8be4cbac74b03b45893465da198d3 2023-04-18 22:54:35,955 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/cfb8be4cbac74b03b45893465da198d3, entries=100, sequenceid=795, filesize=18.6 K 2023-04-18 22:54:35,956 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/11c9909072f844d48e9059e07be395f7 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/11c9909072f844d48e9059e07be395f7 2023-04-18 22:54:35,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681858475950, max timestamp=1681858475957Num uniques 1080; 2023-04-18 22:54:35,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=12820, suffix heap size=140288,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=280 2023-04-18 22:54:35,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681858475950, max timestamp=1681858475957Num uniques 1080; 2023-04-18 22:54:35,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=12820, suffix heap size=140288,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=280 2023-04-18 22:54:35,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681858475950, max timestamp=1681858475957Num uniques 1080; 2023-04-18 22:54:35,960 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=12820, suffix heap size=140288,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=280 2023-04-18 22:54:35,963 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/11c9909072f844d48e9059e07be395f7, entries=100, sequenceid=795, filesize=9.5 K 2023-04-18 22:54:35,965 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/bab32f3b9eb04193ba339dd92b2ec49c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bab32f3b9eb04193ba339dd92b2ec49c 2023-04-18 22:54:35,972 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bab32f3b9eb04193ba339dd92b2ec49c, entries=100, sequenceid=795, filesize=9.5 K 2023-04-18 22:54:35,973 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=73.80 KB/75570 for 052b0c71a65b8786b7851f4ab1dab728 in 953ms, sequenceid=795, compaction requested=false 2023-04-18 22:54:35,973 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:35,973 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:35,973 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:35,974 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:35,974 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:35,974 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:35,974 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:35,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:35,975 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=87.22 KB heapSize=234.19 KB 2023-04-18 22:54:35,975 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:35,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:35,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:35,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:35,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:35,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:35,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:35,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:35,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:35,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:35,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:35,977 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:35,977 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:35,977 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:35,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:35,977 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:35,977 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:35,977 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:35,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:35,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63359a77[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:35,977 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:35,978 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:35,979 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:35,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:35,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:35,979 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:35,979 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:35,979 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:35,979 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:35,980 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:35,981 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:35,981 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:35,981 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:35,981 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:35,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:35,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:35,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:35,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:35,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:35,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:35,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:35,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:35,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:35,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:35,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:35,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:35,984 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:35,984 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:35,984 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:35,984 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:35,984 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:35,987 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:35,987 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:35,987 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:35,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:35,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:35,988 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:35,989 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:35,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:35,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:35,992 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:35,992 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:35,992 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:35,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:35,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:35,992 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:35,992 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:35,992 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:35,993 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:35,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:35,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:35,993 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:35,993 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:35,993 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:35,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:35,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:35,993 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:35,993 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:36,002 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418a98c853f4c394186aaa2ecfc59e0a986 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a98c853f4c394186aaa2ecfc59e0a986 2023-04-18 22:54:36,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:36,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858536000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:36,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858536001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:36,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59026 deadline: 1681858536001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:36,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858536002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,093 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:36,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:36,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:36,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:36,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:36,097 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:36,097 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:36,097 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:36,097 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:36,098 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,098 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,099 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:36,099 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:36,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:36,100 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,100 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:36,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:36,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:36,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:36,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:36,101 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:36,101 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:36,101 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:36,102 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:36,102 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:36,102 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:36,102 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:36,102 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:36,102 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:36,102 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:36,102 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:36,102 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,102 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:36,102 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:36,104 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,104 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,105 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,105 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:36,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:36,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eb36db8[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:36,106 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:36,106 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,108 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,108 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:36,108 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:36,108 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,108 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:36,108 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:36,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:36,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858536105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,110 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:36,110 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:36,110 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:36,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:36,111 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858536109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:36,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858536109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:36,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59026 deadline: 1681858536109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:36,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:36,114 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:36,114 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:36,114 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:36,114 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:36,114 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:36,116 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,116 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,116 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:36,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:36,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:36,116 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,118 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,123 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:36,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:36,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:36,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:36,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:36,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:36,124 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:36,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,124 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:36,124 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:36,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:36,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:36,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,224 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:36,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:36,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:36,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:36,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:36,227 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:36,227 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:36,227 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:36,227 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:36,228 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,228 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,229 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:36,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:36,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:36,229 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,230 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,230 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:36,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:36,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:36,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:36,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:36,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:36,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:36,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:36,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:36,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:36,231 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:36,231 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:36,231 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:36,231 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:36,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:36,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:36,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:36,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:36,233 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:36,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:36,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d02c8e5[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:36,234 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:36,235 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,237 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,237 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:36,237 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:36,237 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:36,237 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,237 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:36,238 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:36,238 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:36,238 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,238 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:36,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:36,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:36,241 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:36,241 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:36,241 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:36,241 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:36,241 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:36,242 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,242 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,242 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:36,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,243 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:36,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:36,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,244 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,246 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,246 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:36,246 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,246 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:36,246 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:36,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:36,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:36,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:36,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:36,246 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,246 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,246 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,246 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,246 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,247 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:36,247 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:36,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:36,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:36,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:36,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858536311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:36,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858536312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:36,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858536313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:36,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59026 deadline: 1681858536313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,347 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:36,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:36,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:36,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:36,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:36,350 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:36,350 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:36,350 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:36,350 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:36,351 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,351 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,353 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:36,353 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:36,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:36,353 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,353 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:36,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:36,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:36,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:36,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:36,354 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:36,354 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:36,354 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:36,355 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:36,355 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:36,355 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:36,355 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:36,355 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:36,355 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:36,355 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:36,355 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:36,355 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,355 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:36,355 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:36,356 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,356 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:36,358 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:36,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ae6c87[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:36,358 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,359 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,359 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:36,359 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:36,359 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:36,359 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,360 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:36,361 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,361 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:36,361 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:36,361 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:36,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:36,363 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:36,363 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:36,363 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:36,363 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:36,363 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:36,366 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,366 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,366 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:36,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,366 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:36,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,367 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:36,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:36,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:36,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:36,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:36,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:36,370 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:36,370 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,370 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,370 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,370 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,370 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,370 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:36,370 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:36,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:36,370 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:36,414 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1151, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/942ab475a76e411f9630cd5f74465ac2 2023-04-18 22:54:36,470 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:36,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:36,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:36,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:36,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:36,474 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:36,474 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:36,474 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:36,474 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:36,474 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,474 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,476 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:36,476 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:36,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:36,476 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,476 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:36,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:36,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:36,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:36,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:36,479 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:36,479 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:36,479 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:36,479 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:36,479 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:36,479 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:36,479 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:36,479 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:36,479 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:36,479 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,479 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:36,479 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:36,481 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,481 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,481 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,481 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:36,482 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:36,483 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f07b6c7[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:36,485 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:36,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:36,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:36,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:36,487 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,487 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:36,487 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,487 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:36,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:36,489 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:36,489 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:36,489 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:36,489 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:36,489 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:36,490 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,490 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,491 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,491 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,492 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,494 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,494 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,494 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:36,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:36,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:36,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:36,494 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:36,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:36,494 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,495 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:36,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:36,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:36,495 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:36,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:36,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,595 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:36,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:36,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:36,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:36,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:36,598 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:36,598 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:36,598 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:36,598 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:36,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,604 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,604 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:36,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:36,604 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,604 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:36,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:36,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:36,605 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:36,605 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:36,605 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:36,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:36,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:36,606 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:36,606 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:36,606 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:36,606 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:36,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:36,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:36,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:36,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:36,607 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,607 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,608 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,608 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:36,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5abcd232[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:36,609 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:36,609 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,610 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,610 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:36,610 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:36,610 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:36,610 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,611 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:36,616 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:36,616 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:36,616 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,616 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:36,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858536616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:36,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858536616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:36,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858536616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:36,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:36,619 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:36,619 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:36,619 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:36,619 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:36,619 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:36,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:36,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59026 deadline: 1681858536617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,620 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,620 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,621 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:36,621 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,622 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,624 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:36,624 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:36,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:36,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:36,624 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:36,624 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,624 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,624 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,624 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,624 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:36,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:36,624 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:36,624 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:36,624 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:36,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:36,625 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:36,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59006 deadline: 1681858536641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,643 DEBUG [Thread-978] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:54:36,725 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:36,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:36,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:36,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:36,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:36,728 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:36,728 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:36,728 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:36,728 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:36,729 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,729 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,730 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:36,730 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:36,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:36,731 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,731 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:36,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:36,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:36,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:36,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:36,732 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:36,732 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:36,732 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:36,736 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:36,736 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:36,736 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:36,736 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:36,736 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:36,736 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:36,736 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:36,736 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:36,736 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,736 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:36,736 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:36,739 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,740 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:36,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21826c60[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:36,743 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:36,743 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,745 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,745 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:36,745 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:36,745 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:36,745 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,745 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:36,747 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:36,747 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:36,747 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:36,747 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:36,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:36,750 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:36,751 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:36,751 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:36,751 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:36,751 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:36,753 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,753 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,753 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:36,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:36,754 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,755 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,759 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,759 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:36,759 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:36,759 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:36,759 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:36,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:36,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:36,759 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,759 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,759 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:36,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:36,759 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:36,759 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,760 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,760 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:36,845 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1151 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/0b68681f16e64401926f4fa83f9adf1a 2023-04-18 22:54:36,860 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:36,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:36,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:36,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:36,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:36,863 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:36,863 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:36,863 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:36,863 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:36,863 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,863 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,865 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:36,865 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:36,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:36,865 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,865 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:36,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:36,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:36,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:36,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:36,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:36,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:36,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:36,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:36,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:36,866 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:36,867 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:36,867 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:36,867 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:36,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:36,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:36,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:36,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:36,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,868 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,869 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,869 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:36,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22cbe683[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:36,873 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:36,873 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,874 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:36,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:36,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:36,874 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,874 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:36,877 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,877 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:36,877 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:36,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,877 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:36,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:36,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:36,879 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:36,879 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:36,879 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:36,879 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:36,879 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:36,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:36,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,882 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:36,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:36,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,883 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,885 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,885 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,885 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,885 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,885 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:36,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:36,885 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:36,885 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:36,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,885 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,885 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:36,885 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:36,885 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:36,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:36,885 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:36,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:36,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:36,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:36,985 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:36,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:36,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:36,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:36,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:36,989 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:36,989 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:36,989 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:36,989 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:36,989 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,989 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,990 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:36,990 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:36,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:36,991 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,991 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:36,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:36,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:36,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:36,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:36,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:36,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:36,992 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:36,992 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:36,992 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:36,992 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:36,992 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:36,992 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:36,992 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:36,992 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:36,992 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:36,992 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:36,993 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:36,993 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,993 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:36,993 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:36,995 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,995 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:36,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:36,995 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:36,995 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:36,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:36,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:36,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:36,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:36,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:36,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:36,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20f8dd93[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:36,997 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:36,997 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,000 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,000 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,000 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:37,000 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:37,000 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:37,000 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:37,002 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,002 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:37,002 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:37,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:37,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:37,002 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:37,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:37,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:37,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:37,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:37,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:37,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:37,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:37,004 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:37,004 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:37,005 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:37,005 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:37,005 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:37,006 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,006 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:37,006 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:37,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:37,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:37,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:37,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:37,007 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:37,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:37,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:37,009 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:37,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:37,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,011 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,012 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:37,012 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:37,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:37,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:37,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:37,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:37,012 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:37,012 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:37,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:37,012 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:37,012 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:37,012 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:37,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:37,012 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,012 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,012 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,012 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,112 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:37,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:37,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:37,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:37,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:37,115 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:37,115 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:37,116 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:37,116 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:37,116 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,116 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:37,117 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,117 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:37,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:37,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:37,118 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,118 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:37,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:37,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:37,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:37,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:37,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:37,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:37,119 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:37,119 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:37,119 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:37,119 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:37,119 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:37,119 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:37,119 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:37,120 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:37,120 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:37,120 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:37,120 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:37,120 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,120 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:37,120 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:37,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:37,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858537120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,121 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:37,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:37,121 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,121 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:37,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:37,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:37,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858537120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ce9a6fc[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:37,123 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:37,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:37,123 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858537120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,124 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:37,124 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,124 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:37,124 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:37,124 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:37,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:37,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59026 deadline: 1681858537124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,127 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:37,127 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,127 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:37,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:37,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:37,127 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:37,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:37,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:37,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:37,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:37,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:37,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:37,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:37,129 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:37,129 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:37,130 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:37,130 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:37,130 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:37,131 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,131 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,131 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:37,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:37,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:37,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,132 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:37,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:37,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:37,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:37,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:37,133 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:37,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:37,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:37,135 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,135 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:37,135 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:37,135 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:37,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:37,135 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:37,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:37,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:37,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:37,135 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:37,135 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:37,135 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:37,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:37,135 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:37,135 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:37,135 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,135 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,235 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:37,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:37,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:37,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:37,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:37,239 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:37,239 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:37,239 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:37,239 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:37,239 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,239 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:37,241 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:37,241 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:37,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:37,241 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,241 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:37,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:37,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:37,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:37,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:37,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:37,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:37,242 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:37,242 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:37,242 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:37,242 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:37,242 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:37,243 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:37,243 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:37,243 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:37,243 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:37,243 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:37,243 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:37,243 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,243 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:37,243 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:37,245 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,245 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:37,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:37,245 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,245 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:37,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:37,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:37,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:37,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:37,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:37,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eba386d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:37,248 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:37,248 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,249 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,249 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:37,249 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,249 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:37,249 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:37,249 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:37,251 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,251 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:37,251 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:37,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:37,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:37,251 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:37,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:37,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:37,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:37,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:37,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:37,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:37,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:37,253 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:37,253 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:37,253 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:37,253 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:37,253 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:37,254 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,255 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:37,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,255 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:37,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:37,255 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:37,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:37,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:37,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:37,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:37,256 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1151 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/09f9927dbdad4f9087d3d8b66bc9a217 2023-04-18 22:54:37,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:37,256 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:37,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:37,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,261 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/942ab475a76e411f9630cd5f74465ac2 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/942ab475a76e411f9630cd5f74465ac2 2023-04-18 22:54:37,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:37,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:37,261 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:37,261 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,261 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:37,261 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:37,261 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:37,261 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:37,261 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,261 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,261 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:37,261 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,265 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/942ab475a76e411f9630cd5f74465ac2, entries=400, sequenceid=1151, filesize=59.8 K 2023-04-18 22:54:37,266 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/0b68681f16e64401926f4fa83f9adf1a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/0b68681f16e64401926f4fa83f9adf1a 2023-04-18 22:54:37,271 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/0b68681f16e64401926f4fa83f9adf1a, entries=150, sequenceid=1151, filesize=11.9 K 2023-04-18 22:54:37,271 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:37,271 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:37,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:37,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:37,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:37,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:37,272 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/09f9927dbdad4f9087d3d8b66bc9a217 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/09f9927dbdad4f9087d3d8b66bc9a217 2023-04-18 22:54:37,276 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/09f9927dbdad4f9087d3d8b66bc9a217, entries=150, sequenceid=1151, filesize=11.9 K 2023-04-18 22:54:37,277 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~254.06 KB/260160, currentSize=87.22 KB/89310 for 052b0c71a65b8786b7851f4ab1dab728 in 1302ms, sequenceid=1151, compaction requested=true 2023-04-18 22:54:37,277 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:37,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:37,277 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:37,277 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:37,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:37,277 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:37,278 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 107082 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:37,278 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34898 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:37,278 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/A is initiating minor compaction (all files) 2023-04-18 22:54:37,278 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/B is initiating minor compaction (all files) 2023-04-18 22:54:37,278 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/A in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:37,278 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/B in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:37,278 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/130e35ea0db64977b8644b85f3de4a4a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/cfb8be4cbac74b03b45893465da198d3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/942ab475a76e411f9630cd5f74465ac2] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=104.6 K 2023-04-18 22:54:37,279 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/d3f6541c5b9c415694b1ef773dba579c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/11c9909072f844d48e9059e07be395f7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/0b68681f16e64401926f4fa83f9adf1a] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=34.1 K 2023-04-18 22:54:37,279 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 130e35ea0db64977b8644b85f3de4a4a, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1681858472468 2023-04-18 22:54:37,279 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting d3f6541c5b9c415694b1ef773dba579c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1681858472468 2023-04-18 22:54:37,279 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting cfb8be4cbac74b03b45893465da198d3, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681858473620 2023-04-18 22:54:37,279 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 11c9909072f844d48e9059e07be395f7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681858473620 2023-04-18 22:54:37,280 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 942ab475a76e411f9630cd5f74465ac2, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=1151, earliestPutTs=1681858475950 2023-04-18 22:54:37,280 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 0b68681f16e64401926f4fa83f9adf1a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1151, earliestPutTs=1681858475950 2023-04-18 22:54:37,289 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#B#compaction#206 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:37,309 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#A#compaction#207 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:37,361 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:37,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:37,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:37,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:37,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:37,365 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:37,365 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:37,365 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:37,365 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:37,365 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,365 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:37,367 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:37,367 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:37,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:37,367 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:37,367 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:37,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:37,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:37,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:37,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:37,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:37,368 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:37,368 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:37,368 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:37,368 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:37,369 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:37,369 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:37,369 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. started... 2023-04-18 22:54:37,369 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-18 22:54:37,369 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:37,369 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:37,369 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:37,369 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:37,369 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:37,369 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:37,711 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/1893f7aedddc4c5c86519d79b690641f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/1893f7aedddc4c5c86519d79b690641f 2023-04-18 22:54:37,715 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/B of 052b0c71a65b8786b7851f4ab1dab728 into 1893f7aedddc4c5c86519d79b690641f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:37,715 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:37,715 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/B, priority=13, startTime=1681858477277; duration=0sec 2023-04-18 22:54:37,716 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:37,716 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:37,718 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34898 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:37,719 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 052b0c71a65b8786b7851f4ab1dab728/C is initiating minor compaction (all files) 2023-04-18 22:54:37,719 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 052b0c71a65b8786b7851f4ab1dab728/C in TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:37,719 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/2957c8377d6741a7afa092c9ed77cf98, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bab32f3b9eb04193ba339dd92b2ec49c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/09f9927dbdad4f9087d3d8b66bc9a217] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp, totalSize=34.1 K 2023-04-18 22:54:37,719 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 2957c8377d6741a7afa092c9ed77cf98, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1681858472468 2023-04-18 22:54:37,720 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting bab32f3b9eb04193ba339dd92b2ec49c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681858473620 2023-04-18 22:54:37,720 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 09f9927dbdad4f9087d3d8b66bc9a217, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1151, earliestPutTs=1681858475950 2023-04-18 22:54:37,728 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 052b0c71a65b8786b7851f4ab1dab728#C#compaction#209 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:37,746 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/e3b60f0ba16646c98adc05cb09978555 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/e3b60f0ba16646c98adc05cb09978555 2023-04-18 22:54:37,763 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/A of 052b0c71a65b8786b7851f4ab1dab728 into e3b60f0ba16646c98adc05cb09978555(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:37,763 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:37,763 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/A, priority=13, startTime=1681858477277; duration=0sec 2023-04-18 22:54:37,763 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:37,779 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/58b0d39fff9743bc9ec94f29d3f84cf2 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/58b0d39fff9743bc9ec94f29d3f84cf2 2023-04-18 22:54:37,787 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 052b0c71a65b8786b7851f4ab1dab728/C of 052b0c71a65b8786b7851f4ab1dab728 into 58b0d39fff9743bc9ec94f29d3f84cf2(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:37,787 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:37,788 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728., storeName=052b0c71a65b8786b7851f4ab1dab728/C, priority=13, startTime=1681858477277; duration=0sec 2023-04-18 22:54:37,788 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:37,788 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418de4745ef03cd46cfaf082c9df20d0e82 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418de4745ef03cd46cfaf082c9df20d0e82 2023-04-18 22:54:38,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:38,126 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. as already flushing 2023-04-18 22:54:38,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:38,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59026 deadline: 1681858538156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:38,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:38,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:38,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858538158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:38,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858538158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:38,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:38,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858538159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:38,202 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1167, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/86a09c58695441aa813c078439af67ce 2023-04-18 22:54:38,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:38,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59026 deadline: 1681858538260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:38,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:38,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858538261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:38,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:38,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858538261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:38,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:38,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858538262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:38,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:38,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59026 deadline: 1681858538463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:38,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:38,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858538465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:38,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:38,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858538465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:38,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:38,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858538465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:38,630 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1167 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/fb80fa1647b74888b0f7969bcf632456 2023-04-18 22:54:38,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:38,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59026 deadline: 1681858538767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:38,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:38,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58994 deadline: 1681858538770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:38,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:38,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58974 deadline: 1681858538770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:38,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:38,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58972 deadline: 1681858538771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:38,981 DEBUG [Thread-993] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a3bb940 to 127.0.0.1:50825 2023-04-18 22:54:38,981 DEBUG [Thread-993] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:38,982 DEBUG [Thread-987] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54451f11 to 127.0.0.1:50825 2023-04-18 22:54:38,982 DEBUG [Thread-989] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c4379c2 to 127.0.0.1:50825 2023-04-18 22:54:38,982 DEBUG [Thread-987] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:38,982 DEBUG [Thread-989] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:38,985 DEBUG [Thread-991] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f43b712 to 127.0.0.1:50825 2023-04-18 22:54:38,985 DEBUG [Thread-991] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:39,042 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1167 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/7214808496ea4ec5a1f2360c0906d0c5 2023-04-18 22:54:39,046 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/86a09c58695441aa813c078439af67ce as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/86a09c58695441aa813c078439af67ce 2023-04-18 22:54:39,049 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/86a09c58695441aa813c078439af67ce, entries=150, sequenceid=1167, filesize=25.4 K 2023-04-18 22:54:39,050 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/fb80fa1647b74888b0f7969bcf632456 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/fb80fa1647b74888b0f7969bcf632456 2023-04-18 22:54:39,053 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/fb80fa1647b74888b0f7969bcf632456, entries=150, sequenceid=1167, filesize=11.9 K 2023-04-18 22:54:39,054 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/7214808496ea4ec5a1f2360c0906d0c5 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/7214808496ea4ec5a1f2360c0906d0c5 2023-04-18 22:54:39,057 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/7214808496ea4ec5a1f2360c0906d0c5, entries=150, sequenceid=1167, filesize=11.9 K 2023-04-18 22:54:39,058 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 052b0c71a65b8786b7851f4ab1dab728 in 1689ms, sequenceid=1167, compaction requested=false 2023-04-18 22:54:39,058 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:39,058 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:39,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:39,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:39,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:39,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:39,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:39,060 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:39,060 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:39,061 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:39,061 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:39,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:39,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:39,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:39,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:39,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:39,062 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:39,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53383cc7[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:39,062 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:39,065 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:39,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:39,065 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:39,065 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:39,065 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:39,065 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:39,066 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:39,069 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:39,069 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:39,069 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:39,069 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:39,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:39,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:39,071 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:39,071 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:39,071 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:39,071 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:39,071 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:39,072 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:39,072 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:39,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:39,072 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:39,073 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:39,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:39,074 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:39,077 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:39,077 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:39,077 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:39,077 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:39,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:39,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:39,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:39,077 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:39,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:39,077 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:39,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:39,077 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:39,077 DEBUG [Thread-986] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:39,077 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:39,077 DEBUG [Thread-986] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:39,077 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:39,077 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:39,177 DEBUG [Thread-986] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:39,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:39,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:39,270 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-18 22:54:39,270 DEBUG [Thread-982] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69f15cb4 to 127.0.0.1:50825 2023-04-18 22:54:39,270 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:39,270 DEBUG [Thread-982] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:39,270 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:39,270 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:39,270 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:39,270 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:39,270 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:39,277 DEBUG [Thread-976] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b3934fb to 127.0.0.1:50825 2023-04-18 22:54:39,277 DEBUG [Thread-976] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:39,279 DEBUG [Thread-980] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1441c9f0 to 127.0.0.1:50825 2023-04-18 22:54:39,279 DEBUG [Thread-980] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:39,280 DEBUG [Thread-984] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c1c92c6 to 127.0.0.1:50825 2023-04-18 22:54:39,280 DEBUG [Thread-984] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:39,683 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418c93d033aaeee418890eaa77873706c76 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c93d033aaeee418890eaa77873706c76 2023-04-18 22:54:40,088 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1187, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/43553f71341f452eb10af4773aabd29f 2023-04-18 22:54:40,497 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1187 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/ba2ad4453d304efca41bccb340e6b58c 2023-04-18 22:54:40,653 DEBUG [Thread-978] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1aa5eef5 to 127.0.0.1:50825 2023-04-18 22:54:40,653 DEBUG [Thread-978] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:40,653 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-18 22:54:40,654 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 132 2023-04-18 22:54:40,654 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 235 2023-04-18 22:54:40,654 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 286 2023-04-18 22:54:40,654 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 308 2023-04-18 22:54:40,654 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 138 2023-04-18 22:54:40,654 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-18 22:54:40,654 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(398): read 4153 2023-04-18 22:54:40,654 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(398): read 4256 2023-04-18 22:54:40,654 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-18 22:54:40,654 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(402): scanned 1775 2023-04-18 22:54:40,654 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(403): verified 5325 rows 2023-04-18 22:54:40,654 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(402): scanned 1759 2023-04-18 22:54:40,654 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(403): verified 5277 rows 2023-04-18 22:54:40,654 INFO [Listener at localhost/36651] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-18 22:54:40,654 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x33c943f0 to 127.0.0.1:50825 2023-04-18 22:54:40,654 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:54:40,656 INFO [Listener at localhost/36651] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-18 22:54:40,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-18 22:54:40,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-18 22:54:40,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-18 22:54:40,663 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858480663"}]},"ts":"1681858480663"} 2023-04-18 22:54:40,664 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-18 22:54:40,666 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-18 22:54:40,666 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=052b0c71a65b8786b7851f4ab1dab728, UNASSIGN}] 2023-04-18 22:54:40,668 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=052b0c71a65b8786b7851f4ab1dab728, UNASSIGN 2023-04-18 22:54:40,668 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=052b0c71a65b8786b7851f4ab1dab728, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:40,668 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858480668"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858480668"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858480668"}]},"ts":"1681858480668"} 2023-04-18 22:54:40,669 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:54:40,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-18 22:54:40,821 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:40,821 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 052b0c71a65b8786b7851f4ab1dab728, disabling compactions & flushes 2023-04-18 22:54:40,821 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:40,907 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1187 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/929aab872f6940abb39e058de3f3a461 2023-04-18 22:54:40,911 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/43553f71341f452eb10af4773aabd29f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/43553f71341f452eb10af4773aabd29f 2023-04-18 22:54:40,914 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/43553f71341f452eb10af4773aabd29f, entries=150, sequenceid=1187, filesize=25.4 K 2023-04-18 22:54:40,915 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/ba2ad4453d304efca41bccb340e6b58c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/ba2ad4453d304efca41bccb340e6b58c 2023-04-18 22:54:40,918 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/ba2ad4453d304efca41bccb340e6b58c, entries=150, sequenceid=1187, filesize=11.9 K 2023-04-18 22:54:40,919 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/929aab872f6940abb39e058de3f3a461 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/929aab872f6940abb39e058de3f3a461 2023-04-18 22:54:40,922 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/929aab872f6940abb39e058de3f3a461, entries=150, sequenceid=1187, filesize=11.9 K 2023-04-18 22:54:40,923 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=26.84 KB/27480 for 052b0c71a65b8786b7851f4ab1dab728 in 1653ms, sequenceid=1187, compaction requested=true 2023-04-18 22:54:40,923 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:40,923 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:40,923 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:40,923 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. because compaction request was cancelled 2023-04-18 22:54:40,923 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:40,923 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. because compaction request was cancelled 2023-04-18 22:54:40,923 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:40,923 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. after waiting 0 ms 2023-04-18 22:54:40,924 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. because compaction request was cancelled 2023-04-18 22:54:40,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:40,924 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:40,924 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 052b0c71a65b8786b7851f4ab1dab728 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-18 22:54:40,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=A 2023-04-18 22:54:40,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:40,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=B 2023-04-18 22:54:40,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:40,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 052b0c71a65b8786b7851f4ab1dab728, store=C 2023-04-18 22:54:40,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:40,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-18 22:54:41,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-18 22:54:41,335 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418e4ad1d8da5bf4ca3b35c48770245effd to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e4ad1d8da5bf4ca3b35c48770245effd 2023-04-18 22:54:41,740 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1194, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/8b7da129987341f4b2a663fe01cd447b 2023-04-18 22:54:41,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-18 22:54:42,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1194 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/e2ffd082f0ed4e9cb42c1ed720cff705 2023-04-18 22:54:42,559 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1194 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/32068e931f4e4584835a6ef0b2ca9ac4 2023-04-18 22:54:42,563 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/A/8b7da129987341f4b2a663fe01cd447b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/8b7da129987341f4b2a663fe01cd447b 2023-04-18 22:54:42,566 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/8b7da129987341f4b2a663fe01cd447b, entries=100, sequenceid=1194, filesize=18.6 K 2023-04-18 22:54:42,567 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/B/e2ffd082f0ed4e9cb42c1ed720cff705 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/e2ffd082f0ed4e9cb42c1ed720cff705 2023-04-18 22:54:42,570 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/e2ffd082f0ed4e9cb42c1ed720cff705, entries=100, sequenceid=1194, filesize=9.5 K 2023-04-18 22:54:42,571 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/.tmp/C/32068e931f4e4584835a6ef0b2ca9ac4 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/32068e931f4e4584835a6ef0b2ca9ac4 2023-04-18 22:54:42,574 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/32068e931f4e4584835a6ef0b2ca9ac4, entries=100, sequenceid=1194, filesize=9.5 K 2023-04-18 22:54:42,575 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 052b0c71a65b8786b7851f4ab1dab728 in 1651ms, sequenceid=1194, compaction requested=true 2023-04-18 22:54:42,597 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/03602746bd4d4194b4394f2e6a67c63b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/483dcda33a47434a9e39b26ac6056208, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/12bdea8a36b149f398169007c86c7cc0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/060aa7cdbdc94d428e4c284d7bb9ff28, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/e84a91fed94046f3a83cdcc598ba04df, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/a06d5ad76ac54652be3830b9eb108fbb, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/b05097add6e849a584b8db900ca1aa4e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/9de05ddfb9b443b6ae95d5fc0c2c0865, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/b4c33416b8b64756a9bac1f3c604779b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/7db9fc67b08a48b788099975584db219, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/664ab986fdf44063a541a43319ba4b44, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/ca5f5c43565044ceaf4c08937d591d3a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/af2ce903d08d4883af4505ae61611ae5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/d37dad4c26964df2909add01b7a32b48, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/cd1d4fdf3ef84db9a1f5863316b8707b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/1fa647f0f4624837950e1ac269bdb4ca, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/c6e259f64b8e4e09931403312eee5a16, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/289036a9cefd4f3094a1417a3caec241, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/98a7e710a80b42df8dd1d180d478da42, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/bd7e82a5aff14f469d1aaace2878b586, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/4d93336e541d4f72acbf42f5d04c1e85, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/d56e9601ec1b437ca90fc773f358fe7d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/77f88f59f7c7425eb6b8688445c8b9e7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/130e35ea0db64977b8644b85f3de4a4a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/cfb8be4cbac74b03b45893465da198d3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/942ab475a76e411f9630cd5f74465ac2] to archive 2023-04-18 22:54:42,598 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:54:42,600 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/03602746bd4d4194b4394f2e6a67c63b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/03602746bd4d4194b4394f2e6a67c63b 2023-04-18 22:54:42,602 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/483dcda33a47434a9e39b26ac6056208 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/483dcda33a47434a9e39b26ac6056208 2023-04-18 22:54:42,603 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/12bdea8a36b149f398169007c86c7cc0 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/12bdea8a36b149f398169007c86c7cc0 2023-04-18 22:54:42,604 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/060aa7cdbdc94d428e4c284d7bb9ff28 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/060aa7cdbdc94d428e4c284d7bb9ff28 2023-04-18 22:54:42,605 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/e84a91fed94046f3a83cdcc598ba04df to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/e84a91fed94046f3a83cdcc598ba04df 2023-04-18 22:54:42,606 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/a06d5ad76ac54652be3830b9eb108fbb to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/a06d5ad76ac54652be3830b9eb108fbb 2023-04-18 22:54:42,608 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/b05097add6e849a584b8db900ca1aa4e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/b05097add6e849a584b8db900ca1aa4e 2023-04-18 22:54:42,609 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/9de05ddfb9b443b6ae95d5fc0c2c0865 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/9de05ddfb9b443b6ae95d5fc0c2c0865 2023-04-18 22:54:42,610 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/b4c33416b8b64756a9bac1f3c604779b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/b4c33416b8b64756a9bac1f3c604779b 2023-04-18 22:54:42,611 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/7db9fc67b08a48b788099975584db219 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/7db9fc67b08a48b788099975584db219 2023-04-18 22:54:42,612 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/664ab986fdf44063a541a43319ba4b44 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/664ab986fdf44063a541a43319ba4b44 2023-04-18 22:54:42,613 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/ca5f5c43565044ceaf4c08937d591d3a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/ca5f5c43565044ceaf4c08937d591d3a 2023-04-18 22:54:42,614 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/af2ce903d08d4883af4505ae61611ae5 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/af2ce903d08d4883af4505ae61611ae5 2023-04-18 22:54:42,615 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/d37dad4c26964df2909add01b7a32b48 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/d37dad4c26964df2909add01b7a32b48 2023-04-18 22:54:42,616 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/cd1d4fdf3ef84db9a1f5863316b8707b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/cd1d4fdf3ef84db9a1f5863316b8707b 2023-04-18 22:54:42,617 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/1fa647f0f4624837950e1ac269bdb4ca to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/1fa647f0f4624837950e1ac269bdb4ca 2023-04-18 22:54:42,618 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/c6e259f64b8e4e09931403312eee5a16 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/c6e259f64b8e4e09931403312eee5a16 2023-04-18 22:54:42,619 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/289036a9cefd4f3094a1417a3caec241 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/289036a9cefd4f3094a1417a3caec241 2023-04-18 22:54:42,620 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/98a7e710a80b42df8dd1d180d478da42 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/98a7e710a80b42df8dd1d180d478da42 2023-04-18 22:54:42,621 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/bd7e82a5aff14f469d1aaace2878b586 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/bd7e82a5aff14f469d1aaace2878b586 2023-04-18 22:54:42,622 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/4d93336e541d4f72acbf42f5d04c1e85 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/4d93336e541d4f72acbf42f5d04c1e85 2023-04-18 22:54:42,623 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/d56e9601ec1b437ca90fc773f358fe7d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/d56e9601ec1b437ca90fc773f358fe7d 2023-04-18 22:54:42,623 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/77f88f59f7c7425eb6b8688445c8b9e7 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/77f88f59f7c7425eb6b8688445c8b9e7 2023-04-18 22:54:42,625 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/130e35ea0db64977b8644b85f3de4a4a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/130e35ea0db64977b8644b85f3de4a4a 2023-04-18 22:54:42,626 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/cfb8be4cbac74b03b45893465da198d3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/cfb8be4cbac74b03b45893465da198d3 2023-04-18 22:54:42,627 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/942ab475a76e411f9630cd5f74465ac2 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/942ab475a76e411f9630cd5f74465ac2 2023-04-18 22:54:42,629 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/0d20ac7e35294aec841e8f25615d59ed, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/e43b786959154461bbdb861f21e3858f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/e326546dc05744f09be9c7f0c7742466, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/01a57e5e38544c68bd248124df3c5e2c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b323964518d94a2e92d4c5ee921a1d8b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b5cd34dc3c3c41bcb5be3cbc442feddd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/5e2f70206e5c422596f05119d7b3f2ae, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/c362bbfc5ac64e7b93ae5f5ecb9bc380, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/83b37d833940414aa1ba0a859242ce09, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/60c1e18f41e845249624c924f50c5351, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/637397546b294984a78a100becc09692, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/a405e11016504bf9924ce7e5c4d7c945, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/d401afac6cd947f28c166c183728ab1e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/090ba75ce0934790acdaa6b3b5f93a1d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/24c3af227571410cb0838dfded867043, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/fe6b94933a664252ab331cd1350b2aef, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/f6603b7a207c4ede9824a79c4d4bc9b8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b560667f507f4cd1b7e84cbb4db294cc, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/1bc5072aa53a4fad8ac6b6ccfe4c15ee, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/4ab1fb5e4657422ca355388d7cf76169, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/ff5cf1b0c1fa4203950beaebccb29856, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/f66707c4967745d188b604a92d21fa1e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/d3f6541c5b9c415694b1ef773dba579c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/97bbe2d797704ac481e4f3b3bc9e7504, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/11c9909072f844d48e9059e07be395f7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/0b68681f16e64401926f4fa83f9adf1a] to archive 2023-04-18 22:54:42,630 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:54:42,632 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/0d20ac7e35294aec841e8f25615d59ed to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/0d20ac7e35294aec841e8f25615d59ed 2023-04-18 22:54:42,633 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/e43b786959154461bbdb861f21e3858f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/e43b786959154461bbdb861f21e3858f 2023-04-18 22:54:42,633 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/e326546dc05744f09be9c7f0c7742466 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/e326546dc05744f09be9c7f0c7742466 2023-04-18 22:54:42,634 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/01a57e5e38544c68bd248124df3c5e2c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/01a57e5e38544c68bd248124df3c5e2c 2023-04-18 22:54:42,635 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b323964518d94a2e92d4c5ee921a1d8b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b323964518d94a2e92d4c5ee921a1d8b 2023-04-18 22:54:42,636 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b5cd34dc3c3c41bcb5be3cbc442feddd to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b5cd34dc3c3c41bcb5be3cbc442feddd 2023-04-18 22:54:42,637 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/5e2f70206e5c422596f05119d7b3f2ae to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/5e2f70206e5c422596f05119d7b3f2ae 2023-04-18 22:54:42,637 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/c362bbfc5ac64e7b93ae5f5ecb9bc380 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/c362bbfc5ac64e7b93ae5f5ecb9bc380 2023-04-18 22:54:42,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/83b37d833940414aa1ba0a859242ce09 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/83b37d833940414aa1ba0a859242ce09 2023-04-18 22:54:42,639 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/60c1e18f41e845249624c924f50c5351 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/60c1e18f41e845249624c924f50c5351 2023-04-18 22:54:42,640 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/637397546b294984a78a100becc09692 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/637397546b294984a78a100becc09692 2023-04-18 22:54:42,640 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/a405e11016504bf9924ce7e5c4d7c945 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/a405e11016504bf9924ce7e5c4d7c945 2023-04-18 22:54:42,641 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/d401afac6cd947f28c166c183728ab1e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/d401afac6cd947f28c166c183728ab1e 2023-04-18 22:54:42,642 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/090ba75ce0934790acdaa6b3b5f93a1d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/090ba75ce0934790acdaa6b3b5f93a1d 2023-04-18 22:54:42,643 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/24c3af227571410cb0838dfded867043 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/24c3af227571410cb0838dfded867043 2023-04-18 22:54:42,644 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/fe6b94933a664252ab331cd1350b2aef to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/fe6b94933a664252ab331cd1350b2aef 2023-04-18 22:54:42,645 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/f6603b7a207c4ede9824a79c4d4bc9b8 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/f6603b7a207c4ede9824a79c4d4bc9b8 2023-04-18 22:54:42,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b560667f507f4cd1b7e84cbb4db294cc to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/b560667f507f4cd1b7e84cbb4db294cc 2023-04-18 22:54:42,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/1bc5072aa53a4fad8ac6b6ccfe4c15ee to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/1bc5072aa53a4fad8ac6b6ccfe4c15ee 2023-04-18 22:54:42,648 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/4ab1fb5e4657422ca355388d7cf76169 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/4ab1fb5e4657422ca355388d7cf76169 2023-04-18 22:54:42,649 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/ff5cf1b0c1fa4203950beaebccb29856 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/ff5cf1b0c1fa4203950beaebccb29856 2023-04-18 22:54:42,650 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/f66707c4967745d188b604a92d21fa1e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/f66707c4967745d188b604a92d21fa1e 2023-04-18 22:54:42,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/d3f6541c5b9c415694b1ef773dba579c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/d3f6541c5b9c415694b1ef773dba579c 2023-04-18 22:54:42,652 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/97bbe2d797704ac481e4f3b3bc9e7504 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/97bbe2d797704ac481e4f3b3bc9e7504 2023-04-18 22:54:42,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/11c9909072f844d48e9059e07be395f7 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/11c9909072f844d48e9059e07be395f7 2023-04-18 22:54:42,654 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/0b68681f16e64401926f4fa83f9adf1a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/0b68681f16e64401926f4fa83f9adf1a 2023-04-18 22:54:42,656 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e251d601ef694fadb8c46576eb9b62cc, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/d82bf489bd1c44dd8b97d3399f346503, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/65ced05abf394198bb4d92512d264758, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/828e6b67ad614ec7b6c7e34e7213e1ec, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/46bc37e2e335411d96b6a28eb47b284f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bb1c64134f12450dad52c348b90e06f2, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/169809fbf04a432090629136ee072d0b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c993cf7cd86342d1b51bee89ea820a4a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/d08ffd7f230448719633aad6388f49e6, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/16f52ded90ed4a3eaafd8d8d751890e4, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/99b4343382ed4895802d70bde55e496d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/7910f6b3d4bc4af4b4200b58c54cee3d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e567d7dc1adf44a7bdc78106dd9730a5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c8013ea5800548dcaf94b4905e9489d7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/5d41c8f2a63c477c8046830184406dc5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bfa538804d1f4edc9cfc443d0d255646, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c30a8162081640d4acac115a00a76ea0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/0a95e4d537434af59ccdbf6cc5bda17d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e75fd267e1794489aae9150761e64c12, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c2ba8162b5de49d8b21aad1998afcf02, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e9aed972cd134fcb8540d1758ea4dfd8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/0e221531b8044cafa1ebd57d2fbac449, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/2957c8377d6741a7afa092c9ed77cf98, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/3543ec11040f40558abe588a5658c415, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bab32f3b9eb04193ba339dd92b2ec49c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/09f9927dbdad4f9087d3d8b66bc9a217] to archive 2023-04-18 22:54:42,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:54:42,658 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e251d601ef694fadb8c46576eb9b62cc to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e251d601ef694fadb8c46576eb9b62cc 2023-04-18 22:54:42,659 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/d82bf489bd1c44dd8b97d3399f346503 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/d82bf489bd1c44dd8b97d3399f346503 2023-04-18 22:54:42,660 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/65ced05abf394198bb4d92512d264758 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/65ced05abf394198bb4d92512d264758 2023-04-18 22:54:42,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/828e6b67ad614ec7b6c7e34e7213e1ec to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/828e6b67ad614ec7b6c7e34e7213e1ec 2023-04-18 22:54:42,662 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/46bc37e2e335411d96b6a28eb47b284f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/46bc37e2e335411d96b6a28eb47b284f 2023-04-18 22:54:42,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bb1c64134f12450dad52c348b90e06f2 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bb1c64134f12450dad52c348b90e06f2 2023-04-18 22:54:42,664 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/169809fbf04a432090629136ee072d0b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/169809fbf04a432090629136ee072d0b 2023-04-18 22:54:42,665 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c993cf7cd86342d1b51bee89ea820a4a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c993cf7cd86342d1b51bee89ea820a4a 2023-04-18 22:54:42,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/d08ffd7f230448719633aad6388f49e6 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/d08ffd7f230448719633aad6388f49e6 2023-04-18 22:54:42,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/16f52ded90ed4a3eaafd8d8d751890e4 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/16f52ded90ed4a3eaafd8d8d751890e4 2023-04-18 22:54:42,668 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/99b4343382ed4895802d70bde55e496d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/99b4343382ed4895802d70bde55e496d 2023-04-18 22:54:42,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/7910f6b3d4bc4af4b4200b58c54cee3d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/7910f6b3d4bc4af4b4200b58c54cee3d 2023-04-18 22:54:42,670 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e567d7dc1adf44a7bdc78106dd9730a5 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e567d7dc1adf44a7bdc78106dd9730a5 2023-04-18 22:54:42,671 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c8013ea5800548dcaf94b4905e9489d7 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c8013ea5800548dcaf94b4905e9489d7 2023-04-18 22:54:42,672 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/5d41c8f2a63c477c8046830184406dc5 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/5d41c8f2a63c477c8046830184406dc5 2023-04-18 22:54:42,673 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bfa538804d1f4edc9cfc443d0d255646 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bfa538804d1f4edc9cfc443d0d255646 2023-04-18 22:54:42,674 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c30a8162081640d4acac115a00a76ea0 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c30a8162081640d4acac115a00a76ea0 2023-04-18 22:54:42,675 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/0a95e4d537434af59ccdbf6cc5bda17d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/0a95e4d537434af59ccdbf6cc5bda17d 2023-04-18 22:54:42,676 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e75fd267e1794489aae9150761e64c12 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e75fd267e1794489aae9150761e64c12 2023-04-18 22:54:42,677 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c2ba8162b5de49d8b21aad1998afcf02 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/c2ba8162b5de49d8b21aad1998afcf02 2023-04-18 22:54:42,678 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e9aed972cd134fcb8540d1758ea4dfd8 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/e9aed972cd134fcb8540d1758ea4dfd8 2023-04-18 22:54:42,679 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/0e221531b8044cafa1ebd57d2fbac449 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/0e221531b8044cafa1ebd57d2fbac449 2023-04-18 22:54:42,680 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/2957c8377d6741a7afa092c9ed77cf98 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/2957c8377d6741a7afa092c9ed77cf98 2023-04-18 22:54:42,681 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/3543ec11040f40558abe588a5658c415 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/3543ec11040f40558abe588a5658c415 2023-04-18 22:54:42,682 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bab32f3b9eb04193ba339dd92b2ec49c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/bab32f3b9eb04193ba339dd92b2ec49c 2023-04-18 22:54:42,683 DEBUG [StoreCloser-TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/09f9927dbdad4f9087d3d8b66bc9a217 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/09f9927dbdad4f9087d3d8b66bc9a217 2023-04-18 22:54:42,687 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/recovered.edits/1197.seqid, newMaxSeqId=1197, maxSeqId=4 2023-04-18 22:54:42,688 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728. 2023-04-18 22:54:42,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 052b0c71a65b8786b7851f4ab1dab728: 2023-04-18 22:54:42,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:42,690 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=052b0c71a65b8786b7851f4ab1dab728, regionState=CLOSED 2023-04-18 22:54:42,690 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858482690"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681858482690"}]},"ts":"1681858482690"} 2023-04-18 22:54:42,693 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-18 22:54:42,693 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 052b0c71a65b8786b7851f4ab1dab728, server=jenkins-hbase4.apache.org,35971,1681858424869 in 2.0220 sec 2023-04-18 22:54:42,694 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-18 22:54:42,694 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=052b0c71a65b8786b7851f4ab1dab728, UNASSIGN in 2.0270 sec 2023-04-18 22:54:42,694 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858482694"}]},"ts":"1681858482694"} 2023-04-18 22:54:42,695 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-18 22:54:42,698 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-18 22:54:42,700 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.0430 sec 2023-04-18 22:54:42,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-18 22:54:42,767 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-18 22:54:42,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-18 22:54:42,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:54:42,770 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:54:42,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-18 22:54:42,770 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:54:42,775 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:42,777 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/recovered.edits] 2023-04-18 22:54:42,780 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/43553f71341f452eb10af4773aabd29f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/43553f71341f452eb10af4773aabd29f 2023-04-18 22:54:42,781 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/86a09c58695441aa813c078439af67ce to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/86a09c58695441aa813c078439af67ce 2023-04-18 22:54:42,782 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/8b7da129987341f4b2a663fe01cd447b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/8b7da129987341f4b2a663fe01cd447b 2023-04-18 22:54:42,783 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/e3b60f0ba16646c98adc05cb09978555 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/A/e3b60f0ba16646c98adc05cb09978555 2023-04-18 22:54:42,785 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/1893f7aedddc4c5c86519d79b690641f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/1893f7aedddc4c5c86519d79b690641f 2023-04-18 22:54:42,786 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/ba2ad4453d304efca41bccb340e6b58c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/ba2ad4453d304efca41bccb340e6b58c 2023-04-18 22:54:42,787 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/e2ffd082f0ed4e9cb42c1ed720cff705 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/e2ffd082f0ed4e9cb42c1ed720cff705 2023-04-18 22:54:42,788 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/fb80fa1647b74888b0f7969bcf632456 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/B/fb80fa1647b74888b0f7969bcf632456 2023-04-18 22:54:42,790 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/32068e931f4e4584835a6ef0b2ca9ac4 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/32068e931f4e4584835a6ef0b2ca9ac4 2023-04-18 22:54:42,791 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/58b0d39fff9743bc9ec94f29d3f84cf2 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/58b0d39fff9743bc9ec94f29d3f84cf2 2023-04-18 22:54:42,792 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/7214808496ea4ec5a1f2360c0906d0c5 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/7214808496ea4ec5a1f2360c0906d0c5 2023-04-18 22:54:42,793 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/929aab872f6940abb39e058de3f3a461 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/C/929aab872f6940abb39e058de3f3a461 2023-04-18 22:54:42,795 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/recovered.edits/1197.seqid to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728/recovered.edits/1197.seqid 2023-04-18 22:54:42,796 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/052b0c71a65b8786b7851f4ab1dab728 2023-04-18 22:54:42,796 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 22:54:42,796 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-18 22:54:42,797 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-18 22:54:42,800 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041805f3e1caa2c445368dc4e7f8b9987a91 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041805f3e1caa2c445368dc4e7f8b9987a91 2023-04-18 22:54:42,801 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041815223920933045df9e4cf0e550ce91f4 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041815223920933045df9e4cf0e550ce91f4 2023-04-18 22:54:42,802 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304182c911c5660ab4a34a930f766c8ebeba8 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304182c911c5660ab4a34a930f766c8ebeba8 2023-04-18 22:54:42,802 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418393a7e820c8f48bd9d29b69968dd36a5 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418393a7e820c8f48bd9d29b69968dd36a5 2023-04-18 22:54:42,803 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041839bd796790ff47cf8e0e02970210ead1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041839bd796790ff47cf8e0e02970210ead1 2023-04-18 22:54:42,804 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304183b20ea261881478499787c947f70cd44 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304183b20ea261881478499787c947f70cd44 2023-04-18 22:54:42,806 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304183f5322708b684f19985815479f20bd2e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304183f5322708b684f19985815479f20bd2e 2023-04-18 22:54:42,807 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418400e8df518f74b35981a1e014479aad3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418400e8df518f74b35981a1e014479aad3 2023-04-18 22:54:42,808 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041848094233ca3b4cf2a80941d91fa5ce55 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041848094233ca3b4cf2a80941d91fa5ce55 2023-04-18 22:54:42,809 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304184974da5b8bd642c6af1c9c8692c80e64 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304184974da5b8bd642c6af1c9c8692c80e64 2023-04-18 22:54:42,810 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304185de41706f24e4e1982513f6d399a38e4 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304185de41706f24e4e1982513f6d399a38e4 2023-04-18 22:54:42,811 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041891be7686d08746d6994ef3a89576b6af to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041891be7686d08746d6994ef3a89576b6af 2023-04-18 22:54:42,812 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304189a43571c215b42dc8f997ddf93632712 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304189a43571c215b42dc8f997ddf93632712 2023-04-18 22:54:42,813 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a98c853f4c394186aaa2ecfc59e0a986 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a98c853f4c394186aaa2ecfc59e0a986 2023-04-18 22:54:42,814 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418b59522e098a84a359766c2bac00542d4 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418b59522e098a84a359766c2bac00542d4 2023-04-18 22:54:42,815 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c26a61658f8145bc8755ddd2328bd00c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c26a61658f8145bc8755ddd2328bd00c 2023-04-18 22:54:42,816 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c27c5d1d519547d4b42c915c2313722a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c27c5d1d519547d4b42c915c2313722a 2023-04-18 22:54:42,817 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c93d033aaeee418890eaa77873706c76 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418c93d033aaeee418890eaa77873706c76 2023-04-18 22:54:42,818 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418dab5d1ecc8ff43988bb4ed083538e566 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418dab5d1ecc8ff43988bb4ed083538e566 2023-04-18 22:54:42,819 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418de4745ef03cd46cfaf082c9df20d0e82 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418de4745ef03cd46cfaf082c9df20d0e82 2023-04-18 22:54:42,820 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418df56147ba7e6490bb751967d8fcd7702 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418df56147ba7e6490bb751967d8fcd7702 2023-04-18 22:54:42,821 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e4ad1d8da5bf4ca3b35c48770245effd to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e4ad1d8da5bf4ca3b35c48770245effd 2023-04-18 22:54:42,822 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e7fee6aa86dd413e9977335c1aaf173e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e7fee6aa86dd413e9977335c1aaf173e 2023-04-18 22:54:42,823 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f1326efe248546efb089cb4336868c8c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f1326efe248546efb089cb4336868c8c 2023-04-18 22:54:42,823 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-18 22:54:42,825 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:54:42,827 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-18 22:54:42,829 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-18 22:54:42,830 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:54:42,830 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-18 22:54:42,830 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681858482830"}]},"ts":"9223372036854775807"} 2023-04-18 22:54:42,831 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-18 22:54:42,832 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 052b0c71a65b8786b7851f4ab1dab728, NAME => 'TestAcidGuarantees,,1681858455826.052b0c71a65b8786b7851f4ab1dab728.', STARTKEY => '', ENDKEY => ''}] 2023-04-18 22:54:42,832 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-18 22:54:42,832 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681858482832"}]},"ts":"9223372036854775807"} 2023-04-18 22:54:42,833 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-18 22:54:42,835 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:54:42,835 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 68 msec 2023-04-18 22:54:42,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-18 22:54:42,871 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-18 22:54:42,884 INFO [Listener at localhost/36651] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=234 (was 233) - Thread LEAK? -, OpenFileDescriptor=565 (was 570), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=409 (was 401) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=6286 (was 6475) 2023-04-18 22:54:42,896 INFO [Listener at localhost/36651] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=234, OpenFileDescriptor=565, MaxFileDescriptor=60000, SystemLoadAverage=409, ProcessCount=169, AvailableMemoryMB=6286 2023-04-18 22:54:42,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 22:54:42,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 22:54:42,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-18 22:54:42,901 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-18 22:54:42,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-18 22:54:42,901 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-18 22:54:42,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-18 22:54:42,902 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:42,903 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa empty. 2023-04-18 22:54:42,903 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:42,903 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 22:54:43,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-18 22:54:43,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-18 22:54:43,312 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 22:54:43,314 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 94dcab632e7fc6a2d56c5605ce1488fa, NAME => 'TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp 2023-04-18 22:54:43,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-18 22:54:43,721 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:54:43,721 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 94dcab632e7fc6a2d56c5605ce1488fa, disabling compactions & flushes 2023-04-18 22:54:43,721 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:43,721 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:43,721 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. after waiting 0 ms 2023-04-18 22:54:43,721 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:43,721 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:43,721 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:43,724 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-18 22:54:43,725 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858483725"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681858483725"}]},"ts":"1681858483725"} 2023-04-18 22:54:43,726 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-18 22:54:43,727 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-18 22:54:43,727 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858483727"}]},"ts":"1681858483727"} 2023-04-18 22:54:43,728 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-18 22:54:43,730 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=94dcab632e7fc6a2d56c5605ce1488fa, ASSIGN}] 2023-04-18 22:54:43,732 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=94dcab632e7fc6a2d56c5605ce1488fa, ASSIGN 2023-04-18 22:54:43,733 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=94dcab632e7fc6a2d56c5605ce1488fa, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35971,1681858424869; forceNewPlan=false, retain=false 2023-04-18 22:54:43,884 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=94dcab632e7fc6a2d56c5605ce1488fa, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:43,884 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858483884"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858483884"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858483884"}]},"ts":"1681858483884"} 2023-04-18 22:54:43,886 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:54:44,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-18 22:54:44,040 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:44,041 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 94dcab632e7fc6a2d56c5605ce1488fa, NAME => 'TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.', STARTKEY => '', ENDKEY => ''} 2023-04-18 22:54:44,041 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:44,041 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:54:44,041 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:44,041 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:44,042 INFO [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:44,043 DEBUG [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A 2023-04-18 22:54:44,043 DEBUG [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A 2023-04-18 22:54:44,044 INFO [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:54:44,044 INFO [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 94dcab632e7fc6a2d56c5605ce1488fa columnFamilyName A 2023-04-18 22:54:44,044 INFO [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] regionserver.HStore(310): Store=94dcab632e7fc6a2d56c5605ce1488fa/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:54:44,044 INFO [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:44,045 DEBUG [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B 2023-04-18 22:54:44,046 DEBUG [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B 2023-04-18 22:54:44,046 INFO [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:54:44,046 INFO [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 94dcab632e7fc6a2d56c5605ce1488fa columnFamilyName B 2023-04-18 22:54:44,046 INFO [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] regionserver.HStore(310): Store=94dcab632e7fc6a2d56c5605ce1488fa/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:54:44,046 INFO [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:44,047 DEBUG [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C 2023-04-18 22:54:44,047 DEBUG [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C 2023-04-18 22:54:44,047 INFO [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:54:44,048 INFO [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 94dcab632e7fc6a2d56c5605ce1488fa columnFamilyName C 2023-04-18 22:54:44,048 INFO [StoreOpener-94dcab632e7fc6a2d56c5605ce1488fa-1] regionserver.HStore(310): Store=94dcab632e7fc6a2d56c5605ce1488fa/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:54:44,048 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:44,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:44,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:44,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 22:54:44,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:44,053 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 22:54:44,053 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 94dcab632e7fc6a2d56c5605ce1488fa; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60957901, jitterRate=-0.09165649116039276}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 22:54:44,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:44,054 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., pid=30, masterSystemTime=1681858484037 2023-04-18 22:54:44,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:44,055 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:44,056 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=94dcab632e7fc6a2d56c5605ce1488fa, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:44,056 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858484056"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681858484056"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681858484056"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681858484056"}]},"ts":"1681858484056"} 2023-04-18 22:54:44,058 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-18 22:54:44,058 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 in 171 msec 2023-04-18 22:54:44,060 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-18 22:54:44,060 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=94dcab632e7fc6a2d56c5605ce1488fa, ASSIGN in 328 msec 2023-04-18 22:54:44,060 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-18 22:54:44,060 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858484060"}]},"ts":"1681858484060"} 2023-04-18 22:54:44,061 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-18 22:54:44,063 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-18 22:54:44,065 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1640 sec 2023-04-18 22:54:45,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-18 22:54:45,007 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-18 22:54:45,007 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x7ed1dde9 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:45,013 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@262f4570, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:45,014 DEBUG [Listener at localhost/36651] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:45,015 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58708, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:45,016 DEBUG [Listener at localhost/36651] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-18 22:54:45,018 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60456, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-18 22:54:45,020 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x70043cf0 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:45,024 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ca898c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:45,024 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x34b7fed0 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:45,027 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6585a70a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:45,028 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x7f4ad99d to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:45,032 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7480f9da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:45,032 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x330b77da to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:45,037 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a24d617, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:45,037 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x7810a231 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:45,041 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@104dbf4e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:45,042 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x753156b2 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:45,046 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@87b3095, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:45,046 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x6dea0f9b to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:45,050 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12115052, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:45,051 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x434e0eab to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:45,056 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e65578e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:45,057 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x5c43dddd to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:45,062 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fcbaba8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:45,062 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x2de064cb to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:54:45,067 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f8ff99f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:54:45,071 DEBUG [hconnection-0x71ec93d4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:45,072 DEBUG [hconnection-0x22b1d81c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:45,072 DEBUG [hconnection-0x2103a780-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:45,072 DEBUG [hconnection-0x5ae59228-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:45,072 DEBUG [hconnection-0x42bb4f50-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:45,072 DEBUG [hconnection-0x1f54f25-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:45,072 DEBUG [hconnection-0x16ba8073-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:45,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:45,073 DEBUG [hconnection-0x1b3465c5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:45,073 DEBUG [hconnection-0x492b5c1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:45,073 DEBUG [hconnection-0x2ed50fb8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:54:45,076 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58720, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:45,076 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58758, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:45,077 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58766, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:45,077 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58790, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:45,085 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58742, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:45,086 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58762, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:45,090 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58776, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:45,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:45,090 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58732, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:45,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:45,092 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:45,092 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:45,092 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58768, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:45,092 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:45,093 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:45,093 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58786, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:54:45,094 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,094 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,099 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:45,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:45,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:45,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,101 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,102 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,102 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:45,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:45,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:45,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:45,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:45,103 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:45,103 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:45,103 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:45,104 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:45,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:45,105 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:45,105 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-18 22:54:45,106 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:54:45,106 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:45,106 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:54:45,106 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:45,106 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:54:45,106 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:45,107 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:45,107 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:45,107 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:45,107 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:45,107 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:45,107 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:45,107 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,107 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:45,107 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:45,114 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,114 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,114 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,114 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:45,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:45,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@354a6776[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:45,117 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:45,117 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,118 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,118 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:45,119 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:45,118 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,119 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:45,119 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:45,120 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:45,120 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:45,120 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,120 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:45,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:45,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:45,123 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:45,123 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:45,123 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:45,123 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:45,123 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:45,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858545121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,125 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,125 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,125 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:45,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,126 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,127 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858545123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858545125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858545125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858545126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,131 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:45,131 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:45,131 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:45,131 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:45,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:45,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:45,131 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,131 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,132 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,132 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/ce117ff783c04a5ca2cb8ecdc4d34379 2023-04-18 22:54:45,132 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:45,132 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:45,132 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:45,132 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,132 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:45,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858545227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858545229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858545230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858545230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858545230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,232 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:45,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:45,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:45,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:45,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:45,235 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:45,235 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:45,235 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:45,235 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:45,235 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,236 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,237 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,237 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:45,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:45,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:45,237 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,237 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:45,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:45,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:45,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:45,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:45,242 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:45,242 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:45,242 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:45,242 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:45,242 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:45,242 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:45,242 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:45,242 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:45,243 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:45,243 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:45,243 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:45,243 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,243 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:45,243 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:45,244 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,244 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,244 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,245 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:45,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6445f965[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:45,246 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:45,246 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,247 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,247 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:45,247 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:45,247 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,247 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:45,247 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:45,255 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,255 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:45,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,255 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:45,255 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:45,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:45,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:45,257 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:45,257 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:45,257 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:45,257 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:45,257 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:45,259 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,259 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,259 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:45,260 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,261 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,263 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,263 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,263 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:45,263 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:45,264 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:45,264 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:45,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:45,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:45,264 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:45,264 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:45,264 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:45,264 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,264 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,264 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,364 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:45,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:45,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:45,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:45,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:45,368 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:45,368 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:45,368 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:45,368 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:45,368 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,368 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,370 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,370 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:45,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:45,370 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,370 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:45,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:45,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:45,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:45,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:45,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:45,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:45,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:45,372 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:45,372 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:45,372 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:45,372 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:45,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:45,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:45,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:45,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:45,375 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,375 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,376 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,376 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:45,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bc3ccd0[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:45,377 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:45,377 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,378 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,379 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:45,379 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:45,379 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,379 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:45,379 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:45,380 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:45,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:45,381 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:45,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:45,383 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:45,383 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:45,383 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:45,383 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:45,383 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:45,384 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,384 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,384 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:45,385 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,386 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,389 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:45,389 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,389 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,389 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,389 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,389 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,389 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,389 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:45,389 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:45,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:45,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:45,389 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:45,389 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:45,390 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:45,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:45,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:45,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858545430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858545430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858545433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858545433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858545433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,490 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:45,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:45,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:45,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:45,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:45,493 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:45,493 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:45,493 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:45,493 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:45,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:45,495 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:45,496 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,496 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:45,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:45,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:45,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:45,497 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:45,497 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:45,497 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:45,497 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:45,497 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:45,497 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:45,497 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:45,497 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:45,497 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:45,497 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:45,497 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:45,498 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,498 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:45,498 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:45,499 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,499 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,499 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,499 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:45,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@98f156b[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:45,501 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:45,501 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,502 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,503 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:45,503 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:45,503 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,503 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:45,503 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:45,505 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,505 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:45,505 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,505 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:45,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:45,507 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:45,508 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:45,508 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:45,508 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:45,508 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:45,509 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,509 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:45,509 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,510 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,511 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,513 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,513 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:45,513 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:45,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:45,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:45,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:45,513 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:45,513 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,513 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,513 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:45,514 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:45,513 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:45,513 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:45,514 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,514 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/5d35722e93944a428fc5ec81efd84cf9 2023-04-18 22:54:45,614 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:45,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:45,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:45,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:45,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:45,618 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:45,618 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:45,618 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:45,618 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:45,618 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,618 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,620 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,620 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:45,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:45,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:45,620 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,620 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:45,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:45,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:45,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:45,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:45,621 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:45,621 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:45,622 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:45,622 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:45,622 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:45,622 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:45,622 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:45,622 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:45,622 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:45,622 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:45,622 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:45,622 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,622 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:45,622 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:45,625 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,625 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,625 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,625 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:45,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f98a2fb[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:45,626 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:45,626 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,628 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:45,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:45,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:45,628 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,628 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:45,630 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:45,630 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:45,630 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,630 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:45,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:45,632 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:45,632 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:45,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:45,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:45,633 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:45,637 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,637 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,638 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:45,638 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,639 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,642 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,642 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,642 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,642 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,642 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,642 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:45,642 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:45,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:45,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:45,642 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:45,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:45,642 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:45,642 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:45,642 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,643 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:45,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:45,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:45,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858545732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858545733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858545735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858545740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:45,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858545740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,743 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:45,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:45,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:45,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:45,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:45,746 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:45,746 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:45,746 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:45,746 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:45,746 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,747 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,748 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:45,748 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:45,748 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,748 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:45,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:45,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:45,749 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:45,750 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:45,750 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:45,750 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:45,750 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:45,750 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:45,750 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:45,750 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:45,750 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:45,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:45,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:45,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:45,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:45,752 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,753 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,754 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,754 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:45,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24dde0bc[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:45,755 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:45,755 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,757 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,758 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:45,758 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:45,758 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:45,758 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,758 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:45,759 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:45,760 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:45,759 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,760 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:45,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:45,762 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:45,762 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:45,762 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:45,762 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:45,762 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:45,764 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,764 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,764 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:45,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:45,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,765 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,767 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,769 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,769 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,769 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,769 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,769 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,769 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,769 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:45,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:45,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:45,769 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,769 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:45,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:45,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:45,769 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:45,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:45,770 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:45,770 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:45,870 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:45,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:45,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:45,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:45,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:45,873 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:45,873 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:45,874 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:45,874 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:45,874 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,874 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,876 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,876 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:45,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:45,877 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,877 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:45,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:45,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:45,878 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:45,878 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:45,878 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:45,878 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:45,880 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,880 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,880 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,880 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:45,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@217d0dbb[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:45,882 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:45,882 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,883 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,883 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:45,883 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:45,883 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:45,883 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,883 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:45,884 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,884 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,885 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:45,885 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:45,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:45,887 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:45,887 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:45,887 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:45,887 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:45,887 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:45,889 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,889 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:45,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,889 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:45,890 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,891 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,893 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:45,894 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,894 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:45,894 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:45,894 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:45,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:45,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:45,894 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:45,894 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:45,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:45,894 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:45,894 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:45,894 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:45,894 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:45,894 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:45,984 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/01ec655a3be64bc2886702010dd49d4f 2023-04-18 22:54:45,989 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/ce117ff783c04a5ca2cb8ecdc4d34379 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/ce117ff783c04a5ca2cb8ecdc4d34379 2023-04-18 22:54:45,994 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/ce117ff783c04a5ca2cb8ecdc4d34379, entries=200, sequenceid=14, filesize=13.9 K 2023-04-18 22:54:45,994 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:45,995 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/5d35722e93944a428fc5ec81efd84cf9 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/5d35722e93944a428fc5ec81efd84cf9 2023-04-18 22:54:45,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:45,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:45,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:45,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:45,999 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:45,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:46,000 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:46,000 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:46,000 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,000 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,001 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,001 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:46,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:46,002 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,002 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:46,003 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/5d35722e93944a428fc5ec81efd84cf9, entries=150, sequenceid=14, filesize=11.6 K 2023-04-18 22:54:46,004 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/01ec655a3be64bc2886702010dd49d4f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/01ec655a3be64bc2886702010dd49d4f 2023-04-18 22:54:46,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:46,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:46,006 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:46,006 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:46,006 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:46,007 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:46,007 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:46,007 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:46,007 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:46,007 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:46,007 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:46,007 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:46,007 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:46,007 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,007 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:46,007 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,009 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,010 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/01ec655a3be64bc2886702010dd49d4f, entries=150, sequenceid=14, filesize=11.6 K 2023-04-18 22:54:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:46,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:46,011 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 94dcab632e7fc6a2d56c5605ce1488fa in 906ms, sequenceid=14, compaction requested=false 2023-04-18 22:54:46,011 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:46,011 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:46,011 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50a860b2[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:46,013 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,013 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:46,013 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:46,013 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:46,013 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,013 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:46,015 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,015 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:46,015 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:46,015 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:46,018 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:46,018 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:46,018 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:46,018 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:46,018 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:46,019 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,019 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:46,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,020 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:46,021 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:46,026 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:46,026 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:46,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:46,026 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:46,026 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:46,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:46,027 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:46,027 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,027 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,027 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,027 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,027 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:46,027 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:46,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:46,027 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,127 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:46,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:46,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:46,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:46,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:46,131 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:46,131 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:46,131 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:46,131 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:46,131 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,131 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,132 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:46,132 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:46,133 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,133 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:46,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:46,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:46,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:46,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:46,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:46,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:46,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:46,134 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:46,134 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:46,134 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 22:54:46,134 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:54:46,135 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:46,135 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:54:46,135 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:46,135 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:54:46,135 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:46,147 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/14a1c6d11aa44a6c89852774a85a2d1c 2023-04-18 22:54:46,162 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/015e9f9241834650bdfd7246c9395519 2023-04-18 22:54:46,178 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/e329576153d24d34a6b3ed96926e4891 2023-04-18 22:54:46,183 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/14a1c6d11aa44a6c89852774a85a2d1c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/14a1c6d11aa44a6c89852774a85a2d1c 2023-04-18 22:54:46,187 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/14a1c6d11aa44a6c89852774a85a2d1c, entries=150, sequenceid=34, filesize=11.6 K 2023-04-18 22:54:46,191 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/015e9f9241834650bdfd7246c9395519 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/015e9f9241834650bdfd7246c9395519 2023-04-18 22:54:46,198 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/015e9f9241834650bdfd7246c9395519, entries=150, sequenceid=34, filesize=11.6 K 2023-04-18 22:54:46,199 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/e329576153d24d34a6b3ed96926e4891 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/e329576153d24d34a6b3ed96926e4891 2023-04-18 22:54:46,203 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/e329576153d24d34a6b3ed96926e4891, entries=150, sequenceid=34, filesize=11.6 K 2023-04-18 22:54:46,204 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 94dcab632e7fc6a2d56c5605ce1488fa in 70ms, sequenceid=34, compaction requested=false 2023-04-18 22:54:46,204 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:46,204 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:46,204 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:46,204 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:46,204 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,204 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:46,204 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:46,206 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,206 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:46,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:46,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:46,206 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,207 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:46,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:46,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a0d8f26[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:46,209 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:46,209 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,210 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,210 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:46,210 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:46,210 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:46,210 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,210 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:46,213 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,213 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:46,213 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:46,213 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:46,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:46,216 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:46,216 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:46,216 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:46,216 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:46,216 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:46,218 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,218 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:46,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,218 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:46,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:46,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:46,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:46,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:46,219 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,220 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,223 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,223 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,223 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:46,223 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,223 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:46,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:46,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:46,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:46,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:46,223 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:46,223 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:46,223 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,223 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,223 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,223 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,223 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:46,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:46,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:46,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:46,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:46,246 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:54:46,248 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:54:46,248 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:46,248 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:54:46,248 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:46,248 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:54:46,248 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:46,257 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-18 22:54:46,264 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/34e0d5d49321438faf642d9050df67b1 2023-04-18 22:54:46,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858546268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858486263, max timestamp=1681858486272Num uniques 859; 2023-04-18 22:54:46,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858486263, max timestamp=1681858486272Num uniques 859; 2023-04-18 22:54:46,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:46,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:46,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858486263, max timestamp=1681858486272Num uniques 859; 2023-04-18 22:54:46,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:54:46,287 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/7141cf04913948e8879f05db024e0c6c 2023-04-18 22:54:46,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858546289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858546291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858546292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858546294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,324 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:46,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:46,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:46,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:46,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:46,327 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:46,327 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:46,327 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:46,327 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:46,327 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,327 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,329 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:46,329 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:46,329 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,329 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:46,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:46,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:46,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:46,330 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:46,330 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:46,330 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:46,331 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:46,331 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:46,331 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:46,331 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:46,331 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:46,331 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:46,331 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:46,331 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:46,331 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,331 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:46,331 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:46,333 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,333 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:46,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:46,334 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,334 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:46,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:46,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:46,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:46,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:46,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@311e48db[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:46,335 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:46,335 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,337 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,337 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:46,337 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:46,337 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:46,337 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,337 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:46,338 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:46,338 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,338 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:46,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:46,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:46,338 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:46,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:46,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:46,340 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:46,341 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:46,341 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:46,341 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:46,341 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:46,342 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,342 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,342 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:46,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:46,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:46,343 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:46,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:46,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:46,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:46,344 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:46,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,347 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,347 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,347 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,347 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:46,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:46,347 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:46,347 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:46,347 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:46,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:46,347 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,347 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,347 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:46,348 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:46,347 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:46,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:46,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:46,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:46,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858546374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858546395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858546395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858546396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858546398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,448 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:46,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:46,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:46,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:46,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:46,451 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:46,451 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:46,451 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:46,451 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:46,452 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,452 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,453 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:46,453 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:46,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:46,454 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,454 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:46,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:46,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:46,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:46,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:46,455 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:46,455 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:46,455 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:46,455 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:46,455 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:46,455 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:46,455 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:46,455 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:46,455 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:46,455 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:46,455 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:46,455 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,455 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:46,455 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:46,457 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,457 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:46,457 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,457 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:46,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:46,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c78b96[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:46,459 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:46,459 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,460 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,460 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:46,460 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:46,460 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,460 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:46,460 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:46,462 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,462 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:46,462 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:46,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:46,462 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:46,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:46,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:46,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:46,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:46,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:46,465 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:46,465 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:46,465 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:46,465 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:46,465 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:46,467 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,467 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:46,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:46,467 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:46,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,467 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:46,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:46,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:46,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:46,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,469 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:46,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,473 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:46,473 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:46,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:46,473 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:46,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:46,473 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:46,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:46,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:46,473 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:46,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:46,474 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,474 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,474 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,474 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,474 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,474 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:46,474 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:46,574 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:46,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:46,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:46,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:46,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:46,578 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:46,578 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:46,578 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:46,578 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:46,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858546577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,578 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,578 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,580 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,580 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:46,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:46,581 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,581 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:46,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:46,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:46,582 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:46,582 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:46,582 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:46,582 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:46,582 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:46,582 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:46,582 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:46,583 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:46,583 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:46,583 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:46,583 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:46,583 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,583 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:46,583 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:46,584 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,584 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:46,585 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,585 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:46,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:46,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bf9c9cc[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:46,586 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:46,586 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,587 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,588 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:46,588 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:46,588 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,588 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:46,588 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:46,589 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:46,589 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:46,589 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,589 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:46,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:46,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:46,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:46,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:46,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:46,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:46,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:46,592 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:46,592 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:46,592 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:46,592 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:46,592 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:46,595 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,595 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,595 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:46,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:46,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:46,595 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:46,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:46,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:46,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:46,597 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:46,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858546598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858546598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858546598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,600 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:46,600 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:46,600 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,600 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:46,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:46,600 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,601 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,601 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,601 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,601 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,601 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:46,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:46,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:46,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:46,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:46,601 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:46,601 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:46,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:46,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858546602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,701 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:46,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:46,703 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/afba6b9c1a2c4bc5bec7643f7cab78c5 2023-04-18 22:54:46,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:46,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:46,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:46,707 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:46,707 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:46,707 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:46,707 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:46,708 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,708 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,709 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:46,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:46,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:46,710 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,710 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/34e0d5d49321438faf642d9050df67b1 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/34e0d5d49321438faf642d9050df67b1 2023-04-18 22:54:46,710 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,710 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:46,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:46,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:46,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:46,711 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:46,712 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:46,712 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:46,712 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:46,712 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:46,712 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:46,712 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:46,712 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:46,712 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:46,712 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:46,712 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:46,712 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,712 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:46,712 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:46,715 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,715 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/34e0d5d49321438faf642d9050df67b1, entries=150, sequenceid=46, filesize=11.6 K 2023-04-18 22:54:46,715 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:46,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:46,715 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,715 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:46,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:46,716 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/7141cf04913948e8879f05db024e0c6c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/7141cf04913948e8879f05db024e0c6c 2023-04-18 22:54:46,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:46,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:46,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:46,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d1b0f2a[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:46,718 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:46,718 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,720 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,720 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:46,720 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:46,720 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:46,721 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/7141cf04913948e8879f05db024e0c6c, entries=150, sequenceid=46, filesize=11.6 K 2023-04-18 22:54:46,721 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/afba6b9c1a2c4bc5bec7643f7cab78c5 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/afba6b9c1a2c4bc5bec7643f7cab78c5 2023-04-18 22:54:46,722 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:46,722 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:46,722 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:46,726 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/afba6b9c1a2c4bc5bec7643f7cab78c5, entries=150, sequenceid=46, filesize=11.6 K 2023-04-18 22:54:46,727 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 94dcab632e7fc6a2d56c5605ce1488fa in 481ms, sequenceid=46, compaction requested=true 2023-04-18 22:54:46,727 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:46,727 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:46,727 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:46,727 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:46,727 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:46,727 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:46,728 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:46,728 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:46,728 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/A is initiating minor compaction (all files) 2023-04-18 22:54:46,728 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/B is initiating minor compaction (all files) 2023-04-18 22:54:46,728 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/A in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:46,728 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/B in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:46,728 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,728 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/5d35722e93944a428fc5ec81efd84cf9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/015e9f9241834650bdfd7246c9395519, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/7141cf04913948e8879f05db024e0c6c] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=34.8 K 2023-04-18 22:54:46,728 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/ce117ff783c04a5ca2cb8ecdc4d34379, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/14a1c6d11aa44a6c89852774a85a2d1c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/34e0d5d49321438faf642d9050df67b1] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=37.1 K 2023-04-18 22:54:46,728 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:46,728 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:46,729 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 5d35722e93944a428fc5ec81efd84cf9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681858485103 2023-04-18 22:54:46,729 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:46,729 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:46,729 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:46,729 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting ce117ff783c04a5ca2cb8ecdc4d34379, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681858485098 2023-04-18 22:54:46,729 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:46,729 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:46,729 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 015e9f9241834650bdfd7246c9395519, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1681858485119 2023-04-18 22:54:46,729 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 14a1c6d11aa44a6c89852774a85a2d1c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1681858485119 2023-04-18 22:54:46,729 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 7141cf04913948e8879f05db024e0c6c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681858486244 2023-04-18 22:54:46,729 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 34e0d5d49321438faf642d9050df67b1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681858486244 2023-04-18 22:54:46,730 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,730 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,730 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:46,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:46,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:46,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:46,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:46,732 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,734 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:46,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:46,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:46,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:46,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,737 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:46,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:46,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:46,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:46,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:46,738 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,738 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:46,738 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:46,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:46,738 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#B#compaction#227 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:46,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:46,741 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#A#compaction#228 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:46,838 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:46,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:46,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:46,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:46,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:46,841 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:46,841 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:46,841 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:46,841 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:46,842 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,842 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,843 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:46,843 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:46,843 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,843 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:46,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:46,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:46,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:46,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:46,844 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:46,844 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:46,844 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:46,845 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:46,845 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:46,845 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:46,845 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:46,845 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-18 22:54:46,845 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:54:46,845 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:46,845 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:54:46,845 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:46,846 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:54:46,846 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:46,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:46,881 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:46,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858546902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858546903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858546904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858546905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:46,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:46,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858546905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:47,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:47,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858547009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:47,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:47,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858547009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:47,155 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/3c0b761459354fcda65b2956a16c2d21 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/3c0b761459354fcda65b2956a16c2d21 2023-04-18 22:54:47,159 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/f92da3cc068240d98a65031c0e765498 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/f92da3cc068240d98a65031c0e765498 2023-04-18 22:54:47,160 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/B of 94dcab632e7fc6a2d56c5605ce1488fa into 3c0b761459354fcda65b2956a16c2d21(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:47,160 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:47,160 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/B, priority=13, startTime=1681858486727; duration=0sec 2023-04-18 22:54:47,161 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:47,161 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:47,162 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:47,163 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/C is initiating minor compaction (all files) 2023-04-18 22:54:47,163 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/C in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:47,163 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/01ec655a3be64bc2886702010dd49d4f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/e329576153d24d34a6b3ed96926e4891, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/afba6b9c1a2c4bc5bec7643f7cab78c5] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=34.8 K 2023-04-18 22:54:47,163 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 01ec655a3be64bc2886702010dd49d4f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681858485103 2023-04-18 22:54:47,164 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting e329576153d24d34a6b3ed96926e4891, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1681858485119 2023-04-18 22:54:47,165 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting afba6b9c1a2c4bc5bec7643f7cab78c5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681858486244 2023-04-18 22:54:47,166 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/A of 94dcab632e7fc6a2d56c5605ce1488fa into f92da3cc068240d98a65031c0e765498(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:47,166 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:47,166 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/A, priority=13, startTime=1681858486727; duration=0sec 2023-04-18 22:54:47,166 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:47,176 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#C#compaction#230 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:47,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:47,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858547211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:47,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:47,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858547212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:47,255 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/03896dbd08724f8abb6ccfb22b3fd219 2023-04-18 22:54:47,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:47,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858547410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:47,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:47,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858547410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:47,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:47,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858547411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:47,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:47,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858547515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:47,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:47,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858547516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:47,586 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/7e53f3ee831344049cca95e9aded315e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/7e53f3ee831344049cca95e9aded315e 2023-04-18 22:54:47,592 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/C of 94dcab632e7fc6a2d56c5605ce1488fa into 7e53f3ee831344049cca95e9aded315e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:47,592 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:47,592 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/C, priority=13, startTime=1681858486727; duration=0sec 2023-04-18 22:54:47,593 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:47,667 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/bdd5f7d4c430428f9908e0dbb03fd167 2023-04-18 22:54:48,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:48,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858548018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:48,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858548019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,079 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/eac0a6b72d954ac48f23ef330d1681b7 2023-04-18 22:54:48,084 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/03896dbd08724f8abb6ccfb22b3fd219 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/03896dbd08724f8abb6ccfb22b3fd219 2023-04-18 22:54:48,088 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/03896dbd08724f8abb6ccfb22b3fd219, entries=150, sequenceid=80, filesize=11.6 K 2023-04-18 22:54:48,089 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/bdd5f7d4c430428f9908e0dbb03fd167 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/bdd5f7d4c430428f9908e0dbb03fd167 2023-04-18 22:54:48,093 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/bdd5f7d4c430428f9908e0dbb03fd167, entries=150, sequenceid=80, filesize=11.6 K 2023-04-18 22:54:48,094 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/eac0a6b72d954ac48f23ef330d1681b7 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/eac0a6b72d954ac48f23ef330d1681b7 2023-04-18 22:54:48,098 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/eac0a6b72d954ac48f23ef330d1681b7, entries=150, sequenceid=80, filesize=11.6 K 2023-04-18 22:54:48,098 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-18 22:54:48,099 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for 94dcab632e7fc6a2d56c5605ce1488fa in 1254ms, sequenceid=80, compaction requested=false 2023-04-18 22:54:48,099 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:48,099 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:48,099 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:48,099 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:48,099 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,104 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:48,104 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:48,105 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,105 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:48,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:48,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:48,106 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:48,106 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:48,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:48,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e80d8c2[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:48,107 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:48,107 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:48,109 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:48,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:48,109 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,109 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:48,112 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,112 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:48,112 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:48,112 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:48,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:48,115 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:48,115 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:48,115 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:48,115 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:48,115 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:48,117 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:48,117 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:48,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:48,117 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:48,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:48,118 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,119 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:48,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:48,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:48,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:48,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:48,122 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:48,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:48,122 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:48,122 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:48,122 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,122 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:48,122 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:48,122 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:48,122 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:48,122 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:48,222 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:48,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:48,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:48,225 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-18 22:54:48,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:48,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:48,226 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:48,226 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:48,226 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:48,226 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:48,226 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:48,226 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:48,232 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:48,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:48,232 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:48,232 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,232 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:48,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:48,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:48,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:48,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:48,234 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:48,234 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:48,234 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:48,250 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:48,250 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:48,250 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:48,250 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:48,251 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-18 22:54:48,251 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:54:48,251 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:48,251 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:54:48,251 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:48,251 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:54:48,251 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:48,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:48,417 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:48,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:48,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858548436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:48,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858548438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:48,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858548439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:48,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858548542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:48,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858548542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:48,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858548542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,660 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/4c54963eaf5143eeae7fc07cb93f3312 2023-04-18 22:54:48,673 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/1640da04082e432fa2aab20f2a4fb77c 2023-04-18 22:54:48,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:48,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858548744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:48,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858548744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:48,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:48,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858548745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:49,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858549023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:49,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858549026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:49,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858549047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:49,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858549048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:49,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858549050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,087 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/470a294393d14f2c8a6e854fdbd19457 2023-04-18 22:54:49,091 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/4c54963eaf5143eeae7fc07cb93f3312 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/4c54963eaf5143eeae7fc07cb93f3312 2023-04-18 22:54:49,095 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/4c54963eaf5143eeae7fc07cb93f3312, entries=150, sequenceid=95, filesize=11.6 K 2023-04-18 22:54:49,096 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/1640da04082e432fa2aab20f2a4fb77c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/1640da04082e432fa2aab20f2a4fb77c 2023-04-18 22:54:49,101 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/1640da04082e432fa2aab20f2a4fb77c, entries=150, sequenceid=95, filesize=11.6 K 2023-04-18 22:54:49,101 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/470a294393d14f2c8a6e854fdbd19457 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/470a294393d14f2c8a6e854fdbd19457 2023-04-18 22:54:49,105 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/470a294393d14f2c8a6e854fdbd19457, entries=150, sequenceid=95, filesize=11.6 K 2023-04-18 22:54:49,106 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 94dcab632e7fc6a2d56c5605ce1488fa in 856ms, sequenceid=95, compaction requested=true 2023-04-18 22:54:49,106 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:49,106 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:49,106 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:49,106 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:49,106 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,106 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:49,106 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:49,108 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:49,108 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:49,108 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,108 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:49,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:49,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:49,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:49,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:49,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:49,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:49,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:49,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7adf7fbf[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:49,110 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:49,110 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:49,111 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:49,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:49,111 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:49,111 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:49,111 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:49,112 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,112 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:49,113 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,113 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:49,113 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:49,113 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:49,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:49,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:49,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:49,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:49,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:49,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:49,116 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:49,116 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:49,116 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:49,116 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:49,116 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:49,119 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,119 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:49,119 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:49,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:49,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:49,120 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:49,121 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,123 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:49,123 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:49,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:49,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:49,123 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,123 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:49,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:49,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:49,123 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:49,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:49,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:49,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:49,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,124 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:49,124 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:49,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:49,224 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:49,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:49,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:49,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:49,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:49,228 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:49,228 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:49,228 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:49,228 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:49,228 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,229 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:49,230 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:49,230 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:49,230 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,230 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:49,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:49,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:49,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:49,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:49,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:49,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:49,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:49,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:49,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:49,232 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:49,232 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:49,232 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 22:54:49,232 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:54:49,232 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:49,232 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:54:49,232 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:49,232 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:54:49,232 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:49,241 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/78b7675f5dc04a4caf2873fb3fb169ea 2023-04-18 22:54:49,258 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/0d1b4120d3ef44d5a6f06140c11afbbd 2023-04-18 22:54:49,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:49,553 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:49,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:49,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858549559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:49,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858549559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:49,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858549561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:49,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858549662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:49,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858549662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:49,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858549664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,670 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/52a9a5ad7ff44025829e8e9d1ba81a3e 2023-04-18 22:54:49,674 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/78b7675f5dc04a4caf2873fb3fb169ea as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/78b7675f5dc04a4caf2873fb3fb169ea 2023-04-18 22:54:49,677 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/78b7675f5dc04a4caf2873fb3fb169ea, entries=150, sequenceid=115, filesize=11.6 K 2023-04-18 22:54:49,678 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/0d1b4120d3ef44d5a6f06140c11afbbd as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/0d1b4120d3ef44d5a6f06140c11afbbd 2023-04-18 22:54:49,682 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/0d1b4120d3ef44d5a6f06140c11afbbd, entries=150, sequenceid=115, filesize=11.6 K 2023-04-18 22:54:49,683 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/52a9a5ad7ff44025829e8e9d1ba81a3e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/52a9a5ad7ff44025829e8e9d1ba81a3e 2023-04-18 22:54:49,686 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/52a9a5ad7ff44025829e8e9d1ba81a3e, entries=150, sequenceid=115, filesize=11.6 K 2023-04-18 22:54:49,687 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 94dcab632e7fc6a2d56c5605ce1488fa in 455ms, sequenceid=115, compaction requested=true 2023-04-18 22:54:49,687 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:49,687 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:49,687 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:49,687 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:49,687 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,687 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:49,687 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:49,689 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,689 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:49,689 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:49,689 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:49,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:49,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fcd7e81[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:49,691 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:49,691 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:49,693 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:49,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:49,693 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,693 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:49,694 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:49,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:49,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:49,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:49,697 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:49,697 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:49,697 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:49,697 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:49,697 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:49,698 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,698 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:49,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,698 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:49,699 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,700 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,702 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,702 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:49,702 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:49,702 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,702 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:49,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:49,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:49,703 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:49,702 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:49,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:49,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:49,703 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:49,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:49,703 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:49,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:49,803 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:49,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:49,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:49,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:49,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:49,807 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:49,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:49,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:49,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:49,808 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,808 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:49,815 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:49,815 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:49,815 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,815 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:49,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:49,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:49,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:49,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:49,816 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:49,817 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:49,817 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:49,817 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:49,817 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:49,817 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:49,817 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:49,817 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-18 22:54:49,817 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:54:49,817 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:49,817 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:54:49,817 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:49,818 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:54:49,818 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:49,826 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/b20df012e0264a84b3dada148d076698 2023-04-18 22:54:49,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:49,867 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:49,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:49,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858549907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:49,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858549910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:49,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:49,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858549911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:50,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858550012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:50,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858550015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:50,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858550015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:50,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858550227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:50,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858550227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:50,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858550228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,237 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/4a5605fac9b0442b98c49c795b322913 2023-04-18 22:54:50,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:50,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858550531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:50,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858550532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:50,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858550533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,647 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/5bfda20667d6409db95e5e01940e9929 2023-04-18 22:54:50,652 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/b20df012e0264a84b3dada148d076698 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/b20df012e0264a84b3dada148d076698 2023-04-18 22:54:50,656 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/b20df012e0264a84b3dada148d076698, entries=150, sequenceid=127, filesize=11.6 K 2023-04-18 22:54:50,657 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/4a5605fac9b0442b98c49c795b322913 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/4a5605fac9b0442b98c49c795b322913 2023-04-18 22:54:50,662 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/4a5605fac9b0442b98c49c795b322913, entries=150, sequenceid=127, filesize=11.6 K 2023-04-18 22:54:50,663 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/5bfda20667d6409db95e5e01940e9929 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/5bfda20667d6409db95e5e01940e9929 2023-04-18 22:54:50,667 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/5bfda20667d6409db95e5e01940e9929, entries=150, sequenceid=127, filesize=11.6 K 2023-04-18 22:54:50,668 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 94dcab632e7fc6a2d56c5605ce1488fa in 851ms, sequenceid=127, compaction requested=true 2023-04-18 22:54:50,668 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:50,668 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:50,668 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:50,668 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:50,668 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,668 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:50,668 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:50,670 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,670 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:50,670 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:50,670 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:50,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:50,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:50,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:50,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a8a480d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:50,672 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:50,672 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:50,674 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:50,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:50,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:50,674 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:50,674 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:50,674 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:50,674 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,674 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:50,675 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:50,675 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,675 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:50,675 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:50,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:50,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:50,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:50,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:50,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:50,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:50,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:50,678 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:50,678 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:50,678 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:50,678 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:50,678 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:50,682 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:50,682 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:50,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:50,683 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:50,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:50,683 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,685 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,687 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,687 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:50,687 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:50,687 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,687 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:50,687 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:50,687 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:50,688 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:50,687 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:50,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:50,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:50,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:50,688 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:50,688 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:50,688 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:50,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:50,788 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:50,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:50,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:50,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:50,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:50,793 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:50,793 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:50,793 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:50,793 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:50,793 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:50,793 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:50,795 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,795 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:50,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:50,795 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:50,795 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:50,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:50,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:50,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:50,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:50,797 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:50,797 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:50,797 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:50,797 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:50,797 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 22:54:50,797 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:54:50,797 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:50,797 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:54:50,797 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:50,797 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:54:50,797 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:50,808 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/0db958221c964a768bb91e69ec6b83aa 2023-04-18 22:54:51,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:51,034 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:51,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858551042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858551043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858551044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,049 DEBUG [Thread-1762] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:54:51,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858551045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858551046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858551147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858551147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858551151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858551151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,218 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/4d9a5afcc90a49659b93293d090927d9 2023-04-18 22:54:51,230 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/54cf765d3dab4c5cb230c068468bf0ba 2023-04-18 22:54:51,233 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/0db958221c964a768bb91e69ec6b83aa as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/0db958221c964a768bb91e69ec6b83aa 2023-04-18 22:54:51,238 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/0db958221c964a768bb91e69ec6b83aa, entries=150, sequenceid=147, filesize=11.8 K 2023-04-18 22:54:51,239 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/4d9a5afcc90a49659b93293d090927d9 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/4d9a5afcc90a49659b93293d090927d9 2023-04-18 22:54:51,242 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/4d9a5afcc90a49659b93293d090927d9, entries=150, sequenceid=147, filesize=11.8 K 2023-04-18 22:54:51,246 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/54cf765d3dab4c5cb230c068468bf0ba as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/54cf765d3dab4c5cb230c068468bf0ba 2023-04-18 22:54:51,250 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/54cf765d3dab4c5cb230c068468bf0ba, entries=150, sequenceid=147, filesize=11.8 K 2023-04-18 22:54:51,251 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 94dcab632e7fc6a2d56c5605ce1488fa in 453ms, sequenceid=147, compaction requested=true 2023-04-18 22:54:51,251 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:51,251 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:51,251 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:51,251 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:51,251 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,256 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:51,256 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:51,261 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,261 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:51,261 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,261 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:51,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:51,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42f4c956[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:51,263 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:51,263 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,264 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,264 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:51,264 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:51,264 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:51,264 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,264 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:51,266 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:51,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:51,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:51,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:51,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:51,268 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:51,268 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:51,268 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:51,268 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:51,268 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:51,270 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,270 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:51,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,270 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:51,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:51,270 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,271 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:51,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:51,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:51,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:51,276 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:51,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:51,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:51,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:51,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,276 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:51,276 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:51,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:51,352 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 22:54:51,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:54:51,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:51,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:54:51,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:51,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:54:51,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:51,360 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/8bd31e02f33b4fe4b83e9b26f17deef5 2023-04-18 22:54:51,374 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/bf341e6c318941d68b309c917f5f9511 2023-04-18 22:54:51,376 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:51,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:51,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858551375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:51,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:51,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:51,381 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:51,381 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:51,381 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:51,381 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:51,381 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,381 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858551378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,383 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:51,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:51,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858551379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858551379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:51,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:51,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:51,385 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:51,385 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:51,385 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:51,385 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:51,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:51,387 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,387 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,387 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:51,388 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,388 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,388 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:51,388 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1476f536[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:51,388 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:51,388 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:51,388 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,389 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,389 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:51,389 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:51,390 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:51,390 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,390 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:51,392 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:51,392 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:51,392 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,392 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:51,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:51,395 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:51,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:51,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:51,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:51,395 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:51,397 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,397 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,397 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:51,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:51,398 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,399 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,401 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:51,401 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:51,401 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:51,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:51,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:51,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:51,401 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,402 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,402 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,402 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,402 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,402 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,402 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:51,402 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:51,402 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:51,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:51,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:51,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858551480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858551483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858551484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858551484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,502 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:51,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:51,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:51,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:51,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:51,506 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:51,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:51,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:51,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:51,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,508 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:51,508 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:51,508 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,508 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:51,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:51,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:51,509 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:51,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:51,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:51,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:51,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:51,510 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:51,510 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:51,510 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:51,510 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:51,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:51,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:51,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:51,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:51,513 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:51,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:51,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:51,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:51,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:51,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43d76430[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:51,515 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:51,515 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,516 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,517 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:51,517 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,517 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:51,517 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:51,517 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:51,518 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:51,518 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:51,518 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,518 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:51,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:51,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:51,521 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:51,521 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:51,521 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:51,521 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:51,521 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:51,522 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,522 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:51,522 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:51,523 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,524 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,527 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,527 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,528 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,528 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:51,528 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:51,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:51,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:51,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:51,528 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,528 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,528 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,528 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,528 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:51,528 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:51,528 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:51,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:51,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:51,628 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:51,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:51,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:51,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:51,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:51,632 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:51,632 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:51,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:51,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:51,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:51,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:51,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,635 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:51,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:51,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:51,636 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:51,636 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:51,636 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:51,636 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:51,637 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:51,637 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,637 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:51,637 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:51,638 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:51,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:51,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:51,640 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:51,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77da23bd[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:51,640 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,641 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:51,641 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,641 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:51,641 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:51,642 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:51,647 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,647 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:51,647 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:51,647 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:51,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:51,649 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:51,649 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:51,649 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:51,649 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:51,649 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:51,651 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,651 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:51,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,651 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:51,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:51,652 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,653 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,655 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,655 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,655 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:51,655 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:51,655 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:51,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:51,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:51,655 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:51,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:51,655 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:51,656 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:51,656 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:51,656 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,656 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,656 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858551683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858551687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858551689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:51,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858551689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,756 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:51,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:51,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:51,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:51,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:51,759 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:51,759 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:51,759 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:51,759 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:51,760 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,760 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,761 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:51,761 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:51,761 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,762 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:51,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:51,763 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:51,763 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:51,763 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:51,763 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:51,766 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f38c76b[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:51,767 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:51,768 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,769 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:51,769 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,769 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:51,771 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:51,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:51,773 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:51,773 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:51,773 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:51,773 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:51,773 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:51,775 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,775 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:51,775 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:51,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:51,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:51,776 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,777 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,781 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,781 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,781 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:51,781 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:51,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:51,781 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:51,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:51,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:51,781 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,781 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:51,781 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,781 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:51,781 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:51,781 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:51,781 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,784 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/3442ea87a15b490cbb6cda78b9588d0b 2023-04-18 22:54:51,788 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/8bd31e02f33b4fe4b83e9b26f17deef5 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8bd31e02f33b4fe4b83e9b26f17deef5 2023-04-18 22:54:51,792 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8bd31e02f33b4fe4b83e9b26f17deef5, entries=200, sequenceid=160, filesize=14.1 K 2023-04-18 22:54:51,792 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/bf341e6c318941d68b309c917f5f9511 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/bf341e6c318941d68b309c917f5f9511 2023-04-18 22:54:51,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/bf341e6c318941d68b309c917f5f9511, entries=150, sequenceid=160, filesize=11.8 K 2023-04-18 22:54:51,797 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/3442ea87a15b490cbb6cda78b9588d0b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/3442ea87a15b490cbb6cda78b9588d0b 2023-04-18 22:54:51,801 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/3442ea87a15b490cbb6cda78b9588d0b, entries=150, sequenceid=160, filesize=11.8 K 2023-04-18 22:54:51,802 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 94dcab632e7fc6a2d56c5605ce1488fa in 450ms, sequenceid=160, compaction requested=true 2023-04-18 22:54:51,802 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:51,802 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:51,802 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:51,802 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-18 22:54:51,802 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-18 22:54:51,802 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:51,805 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83605 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-18 22:54:51,805 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/B is initiating minor compaction (all files) 2023-04-18 22:54:51,805 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/B in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:51,805 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/3c0b761459354fcda65b2956a16c2d21, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/bdd5f7d4c430428f9908e0dbb03fd167, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/1640da04082e432fa2aab20f2a4fb77c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/0d1b4120d3ef44d5a6f06140c11afbbd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/4a5605fac9b0442b98c49c795b322913, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/4d9a5afcc90a49659b93293d090927d9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/bf341e6c318941d68b309c917f5f9511] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=81.6 K 2023-04-18 22:54:51,805 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85995 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-18 22:54:51,806 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/A is initiating minor compaction (all files) 2023-04-18 22:54:51,806 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/A in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:51,806 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/f92da3cc068240d98a65031c0e765498, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/03896dbd08724f8abb6ccfb22b3fd219, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/4c54963eaf5143eeae7fc07cb93f3312, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/78b7675f5dc04a4caf2873fb3fb169ea, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/b20df012e0264a84b3dada148d076698, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/0db958221c964a768bb91e69ec6b83aa, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8bd31e02f33b4fe4b83e9b26f17deef5] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=84.0 K 2023-04-18 22:54:51,807 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 3c0b761459354fcda65b2956a16c2d21, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681858486244 2023-04-18 22:54:51,807 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting f92da3cc068240d98a65031c0e765498, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681858486244 2023-04-18 22:54:51,808 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting bdd5f7d4c430428f9908e0dbb03fd167, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681858486287 2023-04-18 22:54:51,808 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 03896dbd08724f8abb6ccfb22b3fd219, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681858486287 2023-04-18 22:54:51,808 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 1640da04082e432fa2aab20f2a4fb77c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681858486886 2023-04-18 22:54:51,809 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 4c54963eaf5143eeae7fc07cb93f3312, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681858486886 2023-04-18 22:54:51,809 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 0d1b4120d3ef44d5a6f06140c11afbbd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681858488434 2023-04-18 22:54:51,810 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 78b7675f5dc04a4caf2873fb3fb169ea, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681858488434 2023-04-18 22:54:51,810 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 4a5605fac9b0442b98c49c795b322913, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1681858489558 2023-04-18 22:54:51,810 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting b20df012e0264a84b3dada148d076698, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1681858489558 2023-04-18 22:54:51,810 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 4d9a5afcc90a49659b93293d090927d9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1681858489890 2023-04-18 22:54:51,810 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 0db958221c964a768bb91e69ec6b83aa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1681858489890 2023-04-18 22:54:51,810 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting bf341e6c318941d68b309c917f5f9511, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1681858491044 2023-04-18 22:54:51,811 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 8bd31e02f33b4fe4b83e9b26f17deef5, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1681858491041 2023-04-18 22:54:51,825 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#B#compaction#248 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:51,829 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#A#compaction#249 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:51,881 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:51,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:51,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:51,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:51,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:51,885 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:51,885 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:51,885 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:51,885 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:51,885 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,885 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,886 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:51,886 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,890 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:51,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:51,890 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:51,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:51,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:51,891 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:51,891 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:51,891 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:51,892 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:51,892 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:51,892 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:51,892 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:51,892 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 22:54:51,892 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:54:51,892 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:51,892 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:54:51,892 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:51,893 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:54:51,893 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:51,902 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/8801703f3f944450ab672f2f8b11f498 2023-04-18 22:54:51,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:51,991 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:52,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:52,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858552000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:52,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858552001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:52,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858552002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:52,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858552004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:52,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858552105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:52,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858552107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:52,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858552107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:52,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858552107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,239 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/a7b5d8b5fc1a4522936f5abda4100b41 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/a7b5d8b5fc1a4522936f5abda4100b41 2023-04-18 22:54:52,244 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/B of 94dcab632e7fc6a2d56c5605ce1488fa into a7b5d8b5fc1a4522936f5abda4100b41(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:52,245 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:52,245 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/B, priority=9, startTime=1681858491802; duration=0sec 2023-04-18 22:54:52,245 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:52,245 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-18 22:54:52,247 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/03a66cfc2af245d58989e21fc78db7be as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/03a66cfc2af245d58989e21fc78db7be 2023-04-18 22:54:52,247 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83605 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-18 22:54:52,247 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/C is initiating minor compaction (all files) 2023-04-18 22:54:52,247 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/C in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:52,247 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/7e53f3ee831344049cca95e9aded315e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/eac0a6b72d954ac48f23ef330d1681b7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/470a294393d14f2c8a6e854fdbd19457, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/52a9a5ad7ff44025829e8e9d1ba81a3e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/5bfda20667d6409db95e5e01940e9929, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/54cf765d3dab4c5cb230c068468bf0ba, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/3442ea87a15b490cbb6cda78b9588d0b] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=81.6 K 2023-04-18 22:54:52,248 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 7e53f3ee831344049cca95e9aded315e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681858486244 2023-04-18 22:54:52,248 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting eac0a6b72d954ac48f23ef330d1681b7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681858486287 2023-04-18 22:54:52,249 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 470a294393d14f2c8a6e854fdbd19457, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681858486886 2023-04-18 22:54:52,249 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 52a9a5ad7ff44025829e8e9d1ba81a3e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681858488434 2023-04-18 22:54:52,249 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 5bfda20667d6409db95e5e01940e9929, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1681858489558 2023-04-18 22:54:52,250 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 54cf765d3dab4c5cb230c068468bf0ba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1681858489890 2023-04-18 22:54:52,250 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 3442ea87a15b490cbb6cda78b9588d0b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1681858491044 2023-04-18 22:54:52,253 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/A of 94dcab632e7fc6a2d56c5605ce1488fa into 03a66cfc2af245d58989e21fc78db7be(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:52,253 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:52,253 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/A, priority=9, startTime=1681858491802; duration=0sec 2023-04-18 22:54:52,253 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:52,265 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#C#compaction#252 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:52,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:52,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858552309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:52,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858552310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:52,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858552311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:52,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858552311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,318 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/7d2ef07ecad24270bacdd3c3c1b9c8f8 2023-04-18 22:54:52,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:52,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858552614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:52,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:52,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858552614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858552614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:52,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858552614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,675 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/798fa9a66d184e28a2fe5735466a4f86 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/798fa9a66d184e28a2fe5735466a4f86 2023-04-18 22:54:52,679 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/C of 94dcab632e7fc6a2d56c5605ce1488fa into 798fa9a66d184e28a2fe5735466a4f86(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:52,679 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:52,679 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/C, priority=9, startTime=1681858491802; duration=0sec 2023-04-18 22:54:52,679 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:52,729 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/60f4338e5c9d4f0db62f5204589c6c2d 2023-04-18 22:54:52,733 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/8801703f3f944450ab672f2f8b11f498 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8801703f3f944450ab672f2f8b11f498 2023-04-18 22:54:52,736 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8801703f3f944450ab672f2f8b11f498, entries=150, sequenceid=180, filesize=11.8 K 2023-04-18 22:54:52,737 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/7d2ef07ecad24270bacdd3c3c1b9c8f8 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/7d2ef07ecad24270bacdd3c3c1b9c8f8 2023-04-18 22:54:52,740 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/7d2ef07ecad24270bacdd3c3c1b9c8f8, entries=150, sequenceid=180, filesize=11.8 K 2023-04-18 22:54:52,741 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/60f4338e5c9d4f0db62f5204589c6c2d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/60f4338e5c9d4f0db62f5204589c6c2d 2023-04-18 22:54:52,745 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/60f4338e5c9d4f0db62f5204589c6c2d, entries=150, sequenceid=180, filesize=11.8 K 2023-04-18 22:54:52,745 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 94dcab632e7fc6a2d56c5605ce1488fa in 853ms, sequenceid=180, compaction requested=false 2023-04-18 22:54:52,746 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:52,746 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:52,746 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:52,746 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:52,746 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,746 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:52,746 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:52,747 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,747 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:52,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:52,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:52,748 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:52,748 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:52,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:52,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b620ba3[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:52,749 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:52,750 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:52,751 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:52,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:52,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:52,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:52,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:52,751 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,751 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:52,754 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:52,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:52,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:52,758 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:52,758 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:52,758 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:52,758 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:52,758 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:52,760 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:52,760 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:52,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:52,760 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:52,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:52,761 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:52,762 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,765 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,765 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:52,765 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:52,765 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:52,765 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:52,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:52,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:52,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:52,765 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:52,765 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,766 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:52,766 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:52,766 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:52,766 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:52,766 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:52,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:52,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:52,866 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:52,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:52,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:52,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:52,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:52,870 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:52,870 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:52,870 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:52,870 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:52,870 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:52,870 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:52,871 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:52,871 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:52,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:52,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:52,873 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:52,873 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:52,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:52,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:52,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:52,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:52,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:52,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:52,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:52,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:52,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:52,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:52,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:52,874 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:52,874 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:52,874 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-18 22:54:52,875 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:54:52,875 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:52,875 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:54:52,875 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:52,875 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:54:52,875 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:52,883 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/1e604fc9d4a149cea0690af542a248db 2023-04-18 22:54:52,902 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/50dc910baf3943bb8bd4457a48215f64 2023-04-18 22:54:53,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:53,123 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:53,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858553134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858553135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858553136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858553136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858553238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858553239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858553239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858553239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,319 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/94127e0d75f442bd85728934f6bc520b 2023-04-18 22:54:53,323 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/1e604fc9d4a149cea0690af542a248db as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/1e604fc9d4a149cea0690af542a248db 2023-04-18 22:54:53,337 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/1e604fc9d4a149cea0690af542a248db, entries=150, sequenceid=195, filesize=11.8 K 2023-04-18 22:54:53,338 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/50dc910baf3943bb8bd4457a48215f64 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/50dc910baf3943bb8bd4457a48215f64 2023-04-18 22:54:53,342 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/50dc910baf3943bb8bd4457a48215f64, entries=150, sequenceid=195, filesize=11.8 K 2023-04-18 22:54:53,343 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/94127e0d75f442bd85728934f6bc520b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/94127e0d75f442bd85728934f6bc520b 2023-04-18 22:54:53,346 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/94127e0d75f442bd85728934f6bc520b, entries=150, sequenceid=195, filesize=11.8 K 2023-04-18 22:54:53,347 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 94dcab632e7fc6a2d56c5605ce1488fa in 473ms, sequenceid=195, compaction requested=true 2023-04-18 22:54:53,347 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:53,347 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:53,347 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:53,347 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:53,347 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,347 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:53,347 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:53,349 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,349 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:53,349 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,349 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:53,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:53,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@436c0e1b[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:53,350 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:53,350 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,352 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:53,352 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,352 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:53,353 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,353 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:53,353 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:53,353 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:53,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:53,356 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:53,356 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:53,356 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:53,356 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:53,356 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:53,365 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,365 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:53,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,365 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:53,365 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:53,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:53,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,366 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:53,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:53,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:53,369 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:53,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:53,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:53,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,369 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:53,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:53,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,369 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:53,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:53,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:53,443 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-18 22:54:53,444 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:54:53,444 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:53,444 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:54:53,444 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:53,444 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:54:53,444 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:53,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858553449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858553450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858553451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858553451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,469 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:53,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:53,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:53,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:53,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:53,473 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:53,473 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:53,473 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:53,473 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:53,473 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,473 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,475 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:53,475 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:53,475 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,475 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:53,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:53,476 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:53,477 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:53,477 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:53,477 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:53,477 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:53,477 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:53,477 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:53,477 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:53,477 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:53,477 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:53,477 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:53,477 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,477 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:53,478 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:53,479 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,479 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:53,480 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,480 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:53,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:53,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@650d0a00[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:53,481 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:53,481 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,484 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,484 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:53,484 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:53,484 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:53,484 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,484 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:53,486 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,486 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:53,486 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:53,486 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:53,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:53,489 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:53,489 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:53,489 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:53,489 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:53,489 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:53,491 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,491 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:53,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,491 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:53,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:53,491 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:53,492 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:53,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:53,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:53,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:53,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:53,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:53,495 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:53,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:53,495 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:53,496 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:53,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:53,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:53,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858553551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858553553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858553554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858553555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,596 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:53,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:53,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:53,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:53,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:53,599 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:53,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:53,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:53,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:53,600 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,600 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,602 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,602 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:53,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:53,602 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,602 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:53,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:53,604 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:53,604 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:53,604 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:53,604 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:53,606 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:53,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:53,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:53,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:53,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c978a55[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:53,608 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:53,608 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,609 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:53,609 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,609 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:53,610 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:53,611 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:53,611 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:53,611 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:53,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:53,613 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:53,613 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:53,613 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:53,613 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:53,613 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:53,616 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,616 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:53,616 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,617 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,618 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:53,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,620 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:53,620 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,620 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:53,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:53,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:53,620 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,620 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:53,620 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:53,620 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:53,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:53,620 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:53,621 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,621 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,621 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,621 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:53,720 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:53,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:53,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:53,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:53,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:53,724 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:53,724 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:53,724 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:53,724 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:53,724 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,724 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,725 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:53,725 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:53,726 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,726 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:53,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:53,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:53,727 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:53,727 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:53,727 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:53,727 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:53,727 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:53,727 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:53,727 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:53,727 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:53,727 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:53,727 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:53,728 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:53,729 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,729 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:53,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:53,729 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,729 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:53,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5703a977[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:53,731 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:53,731 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,733 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:53,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:53,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:53,734 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,734 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:53,735 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,735 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:53,735 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:53,735 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:53,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:53,738 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:53,738 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:53,738 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:53,738 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:53,738 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:53,739 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,739 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:53,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,739 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:53,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:53,740 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,741 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858553753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,755 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:53,755 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:53,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:53,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:53,756 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:53,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:53,756 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:53,756 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,756 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,756 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,756 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,756 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,756 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,756 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:53,756 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:53,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:53,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858553756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858553757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:53,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858553758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,852 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/e888b495ecf64b279bbe64d4a00fc9cb 2023-04-18 22:54:53,856 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:53,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:53,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:53,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:53,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:53,860 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:53,860 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:53,860 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:53,860 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:53,860 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,860 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,862 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,862 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:53,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,862 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,862 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:53,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:53,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:53,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:53,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:53,863 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:53,863 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:53,863 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:53,864 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:53,864 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:53,864 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:53,864 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:53,864 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:53,866 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:53,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:53,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:53,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@743c795a[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:53,868 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:53,868 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,870 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:53,870 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,870 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:53,873 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:53,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:53,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:53,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:53,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:53,875 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:53,875 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:53,875 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:53,875 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:53,875 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:53,876 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,876 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:53,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,876 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:53,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:53,877 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,878 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,880 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,880 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:53,880 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:53,880 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,880 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:53,880 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,880 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:53,880 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:53,880 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:53,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:53,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:53,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:53,881 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:53,881 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:53,981 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:53,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:53,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:53,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:53,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:53,985 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:53,985 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:53,985 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:53,985 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:53,985 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,985 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,986 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,986 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:53,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:53,987 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,987 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:53,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:53,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:53,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:53,988 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:53,988 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:53,988 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:53,988 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:53,988 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,989 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:53,989 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:53,991 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,991 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:53,992 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,992 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:53,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:53,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79ec30d7[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:53,993 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:53,993 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,995 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:53,995 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,995 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:53,996 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,996 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:53,996 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,996 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:53,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:53,999 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:53,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:53,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:53,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:53,999 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:54,000 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,001 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:54,000 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,001 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:54,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,002 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:54,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,005 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,005 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,005 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,005 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:54,005 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:54,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:54,005 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,005 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,005 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:54,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:54,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:54,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:54,005 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,006 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:54,006 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:54,006 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:54,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:54,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858554058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:54,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858554060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:54,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858554061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:54,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858554062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,106 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:54,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:54,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:54,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:54,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:54,109 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:54,110 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:54,110 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:54,110 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:54,110 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,110 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,111 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,111 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:54,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:54,112 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,112 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:54,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:54,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:54,113 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:54,113 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:54,113 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:54,113 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:54,113 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:54,113 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:54,113 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:54,113 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:54,113 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:54,113 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:54,113 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:54,113 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,114 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:54,114 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:54,115 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:54,115 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:54,115 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,115 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:54,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:54,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:54,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:54,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39e9d89d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:54,117 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:54,117 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,118 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,118 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:54,118 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,118 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:54,118 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:54,118 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:54,120 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:54,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:54,123 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:54,123 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:54,123 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:54,123 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:54,124 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:54,125 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,125 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:54,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,125 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,126 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:54,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,127 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,129 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,129 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,129 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,129 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,129 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,129 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:54,129 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:54,129 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:54,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:54,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:54,129 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,129 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:54,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:54,130 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:54,130 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:54,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:54,230 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:54,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:54,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:54,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:54,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:54,233 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:54,233 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:54,234 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:54,234 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:54,234 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,234 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,238 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:54,238 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:54,239 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,239 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:54,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:54,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:54,240 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:54,240 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:54,240 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:54,240 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:54,242 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,242 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:54,242 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,242 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@736b96c6[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:54,244 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:54,244 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,245 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:54,245 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,245 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:54,247 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:54,247 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:54,247 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,247 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:54,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:54,249 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:54,249 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:54,249 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:54,249 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:54,249 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:54,255 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,255 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,255 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:54,255 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,257 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,259 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:54,260 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:54,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:54,260 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:54,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:54,260 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:54,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:54,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:54,260 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,260 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,260 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,260 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,260 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,260 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,260 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:54,260 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:54,265 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/b7c024cc3d3d4bb0a7a0af6a6d8c66dd 2023-04-18 22:54:54,360 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:54,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:54,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:54,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:54,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:54,364 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:54,364 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:54,364 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:54,364 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:54,364 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,364 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,366 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:54,366 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:54,366 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,366 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:54,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:54,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:54,368 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:54,368 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:54,368 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:54,368 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:54,368 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:54,368 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:54,368 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:54,368 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:54,368 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,368 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:54,368 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:54,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,369 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:54,370 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,370 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@648ac758[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:54,371 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:54,371 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,374 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,374 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:54,374 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:54,374 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,374 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:54,374 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:54,375 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,375 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:54,375 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:54,375 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:54,377 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:54,378 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:54,378 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:54,378 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:54,378 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:54,379 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,379 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:54,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,379 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:54,379 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:54,380 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,382 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,383 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,383 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,382 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:54,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:54,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:54,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:54,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:54,383 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:54,383 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,383 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:54,383 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:54,383 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:54,383 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:54,383 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,383 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:54,483 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:54,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:54,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:54,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:54,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:54,487 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:54,487 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:54,487 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:54,487 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:54,487 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,487 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,490 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:54,490 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:54,490 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,490 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:54,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:54,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:54,491 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:54,491 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:54,491 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:54,491 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:54,492 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:54,492 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:54,492 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:54,492 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:54,492 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:54,493 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,493 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:54,494 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,494 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:54,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:54,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cf82996[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:54,495 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:54,495 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,497 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,497 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:54,497 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:54,497 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,497 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:54,497 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:54,498 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:54,498 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:54,499 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:54,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:54,501 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:54,501 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:54,501 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:54,501 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:54,501 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:54,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:54,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:54,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:54,505 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:54,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,508 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,509 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,509 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,509 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:54,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:54,509 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:54,509 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,509 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:54,509 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:54,509 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:54,509 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:54,509 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:54,509 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:54,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:54,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858554561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:54,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858554564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:54,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858554566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:54,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858554568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,609 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:54,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:54,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:54,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:54,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:54,612 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:54,612 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:54,612 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:54,612 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:54,612 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,613 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,614 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:54,614 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:54,614 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,614 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:54,615 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:54,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:54,615 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:54,615 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:54,616 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:54,616 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:54,621 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,621 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:54,621 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,621 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:54,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:54,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b223ba4[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:54,623 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:54,623 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,624 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,624 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:54,624 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:54,624 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:54,624 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:54,626 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:54,626 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:54,626 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,626 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:54,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:54,628 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:54,628 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:54,628 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:54,628 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:54,628 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:54,630 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,630 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:54,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,630 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,630 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:54,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:54,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:54,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:54,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:54,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:54,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:54,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:54,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:54,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,635 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:54,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:54,635 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:54,635 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,635 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:54,635 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:54,635 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,674 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/54f3af4c0e5045f6b4c4910e217ce1d5 2023-04-18 22:54:54,679 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/e888b495ecf64b279bbe64d4a00fc9cb as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/e888b495ecf64b279bbe64d4a00fc9cb 2023-04-18 22:54:54,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/e888b495ecf64b279bbe64d4a00fc9cb, entries=150, sequenceid=216, filesize=11.8 K 2023-04-18 22:54:54,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/b7c024cc3d3d4bb0a7a0af6a6d8c66dd as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/b7c024cc3d3d4bb0a7a0af6a6d8c66dd 2023-04-18 22:54:54,687 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/b7c024cc3d3d4bb0a7a0af6a6d8c66dd, entries=150, sequenceid=216, filesize=11.8 K 2023-04-18 22:54:54,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/54f3af4c0e5045f6b4c4910e217ce1d5 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/54f3af4c0e5045f6b4c4910e217ce1d5 2023-04-18 22:54:54,691 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/54f3af4c0e5045f6b4c4910e217ce1d5, entries=150, sequenceid=216, filesize=11.8 K 2023-04-18 22:54:54,692 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 94dcab632e7fc6a2d56c5605ce1488fa in 1249ms, sequenceid=216, compaction requested=true 2023-04-18 22:54:54,692 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:54,692 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:54,692 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:54:54,692 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:54,692 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:54:54,692 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:54,693 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:54:54,693 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:54:54,693 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/A is initiating minor compaction (all files) 2023-04-18 22:54:54,693 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/B is initiating minor compaction (all files) 2023-04-18 22:54:54,693 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/A in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:54,693 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/B in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:54,693 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/03a66cfc2af245d58989e21fc78db7be, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8801703f3f944450ab672f2f8b11f498, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/1e604fc9d4a149cea0690af542a248db, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/e888b495ecf64b279bbe64d4a00fc9cb] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=47.3 K 2023-04-18 22:54:54,693 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/a7b5d8b5fc1a4522936f5abda4100b41, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/7d2ef07ecad24270bacdd3c3c1b9c8f8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/50dc910baf3943bb8bd4457a48215f64, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/b7c024cc3d3d4bb0a7a0af6a6d8c66dd] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=47.3 K 2023-04-18 22:54:54,694 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 03a66cfc2af245d58989e21fc78db7be, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1681858491044 2023-04-18 22:54:54,694 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting a7b5d8b5fc1a4522936f5abda4100b41, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1681858491044 2023-04-18 22:54:54,694 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 8801703f3f944450ab672f2f8b11f498, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681858491373 2023-04-18 22:54:54,694 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 7d2ef07ecad24270bacdd3c3c1b9c8f8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681858491373 2023-04-18 22:54:54,695 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 50dc910baf3943bb8bd4457a48215f64, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1681858491995 2023-04-18 22:54:54,695 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 1e604fc9d4a149cea0690af542a248db, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1681858491995 2023-04-18 22:54:54,695 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting b7c024cc3d3d4bb0a7a0af6a6d8c66dd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681858493131 2023-04-18 22:54:54,695 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting e888b495ecf64b279bbe64d4a00fc9cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681858493131 2023-04-18 22:54:54,703 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#A#compaction#261 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:54,703 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#B#compaction#260 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:54,735 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:54,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:54,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:54,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:54,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:54,738 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:54,738 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:54,738 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:54,738 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:54,739 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,739 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,740 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:54,740 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:54,740 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:54,741 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:54,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:54,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:54,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:54,742 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:54,742 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:54,742 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-18 22:54:54,742 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:54:54,742 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:54,743 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:54:54,743 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:54,743 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:54:54,743 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:54,749 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/918c6daa773f4f039352864a71874aa4 2023-04-18 22:54:54,761 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/83d5b7b2318047618188f488e5ba40df 2023-04-18 22:54:55,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:55,077 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:55,112 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/d22e0e0335d846d6ad3c0fb57044b818 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/d22e0e0335d846d6ad3c0fb57044b818 2023-04-18 22:54:55,117 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/690477a63cc647d695d2898ff398bbe3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/690477a63cc647d695d2898ff398bbe3 2023-04-18 22:54:55,117 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/B of 94dcab632e7fc6a2d56c5605ce1488fa into d22e0e0335d846d6ad3c0fb57044b818(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:55,117 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:55,117 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/B, priority=12, startTime=1681858494692; duration=0sec 2023-04-18 22:54:55,118 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:55,118 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:54:55,119 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:54:55,119 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/C is initiating minor compaction (all files) 2023-04-18 22:54:55,119 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/C in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:55,119 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/798fa9a66d184e28a2fe5735466a4f86, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/60f4338e5c9d4f0db62f5204589c6c2d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/94127e0d75f442bd85728934f6bc520b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/54f3af4c0e5045f6b4c4910e217ce1d5] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=47.3 K 2023-04-18 22:54:55,120 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 798fa9a66d184e28a2fe5735466a4f86, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1681858491044 2023-04-18 22:54:55,120 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 60f4338e5c9d4f0db62f5204589c6c2d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681858491373 2023-04-18 22:54:55,121 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 94127e0d75f442bd85728934f6bc520b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1681858491995 2023-04-18 22:54:55,121 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 54f3af4c0e5045f6b4c4910e217ce1d5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681858493131 2023-04-18 22:54:55,124 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/A of 94dcab632e7fc6a2d56c5605ce1488fa into 690477a63cc647d695d2898ff398bbe3(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:55,124 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:55,124 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/A, priority=12, startTime=1681858494692; duration=0sec 2023-04-18 22:54:55,124 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:55,127 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#C#compaction#265 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:55,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858495095, max timestamp=1681858495135Num uniques 859; 2023-04-18 22:54:55,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:54:55,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858495095, max timestamp=1681858495135Num uniques 859; 2023-04-18 22:54:55,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:54:55,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858495095, max timestamp=1681858495135Num uniques 859; 2023-04-18 22:54:55,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:54:55,142 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/cbb51ea96ec34ab393007f8bbc3ea413 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/cbb51ea96ec34ab393007f8bbc3ea413 2023-04-18 22:54:55,146 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/C of 94dcab632e7fc6a2d56c5605ce1488fa into cbb51ea96ec34ab393007f8bbc3ea413(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:55,146 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:55,146 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/C, priority=12, startTime=1681858494692; duration=0sec 2023-04-18 22:54:55,146 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:55,170 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/935dd618655d4d74862b8195b3b4ef9d 2023-04-18 22:54:55,174 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/918c6daa773f4f039352864a71874aa4 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/918c6daa773f4f039352864a71874aa4 2023-04-18 22:54:55,178 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/918c6daa773f4f039352864a71874aa4, entries=150, sequenceid=227, filesize=11.8 K 2023-04-18 22:54:55,178 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/83d5b7b2318047618188f488e5ba40df as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/83d5b7b2318047618188f488e5ba40df 2023-04-18 22:54:55,182 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/83d5b7b2318047618188f488e5ba40df, entries=150, sequenceid=227, filesize=11.8 K 2023-04-18 22:54:55,183 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/935dd618655d4d74862b8195b3b4ef9d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/935dd618655d4d74862b8195b3b4ef9d 2023-04-18 22:54:55,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:55,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858555182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,186 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/935dd618655d4d74862b8195b3b4ef9d, entries=150, sequenceid=227, filesize=11.8 K 2023-04-18 22:54:55,187 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 94dcab632e7fc6a2d56c5605ce1488fa in 445ms, sequenceid=227, compaction requested=false 2023-04-18 22:54:55,187 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:55,187 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:55,187 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:55,189 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,189 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,190 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,190 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:55,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24d051a0[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:55,191 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:55,191 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,192 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,193 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:55,193 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:55,193 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:55,193 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,193 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:55,195 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,195 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:55,195 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:55,195 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:55,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:55,197 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:55,197 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:55,197 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:55,197 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:55,198 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:55,199 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,199 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,199 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:55,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:55,200 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,201 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,204 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,204 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:55,204 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:55,204 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,204 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,204 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,204 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,204 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,204 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:55,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:55,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:55,204 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:55,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:55,204 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:55,204 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:55,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:55,289 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-18 22:54:55,289 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:54:55,289 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:55,290 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:54:55,290 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:55,290 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:54:55,290 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:54:55,304 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:55,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:55,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:55,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:55,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:55,308 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:55,308 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:55,308 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:55,308 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:55,308 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,308 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,309 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,309 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:55,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:55,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:55,310 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,310 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:55,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:55,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:55,311 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:55,311 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:55,311 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:55,311 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:55,311 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:55,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:55,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858555312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,317 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,317 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,317 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,317 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:55,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1652f265[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:55,318 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:55,318 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,320 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,320 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:55,320 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:55,320 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:55,320 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,320 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:55,321 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:55,321 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,321 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,322 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:55,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:55,324 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:55,324 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:55,324 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:55,324 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:55,324 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:55,326 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,326 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,326 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,326 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,327 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,331 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,331 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:55,331 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:55,331 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:55,331 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:55,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:55,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:55,331 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,331 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,331 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,331 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,331 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,331 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:55,331 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:55,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:55,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:55,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858555416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,431 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:55,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:55,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:55,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:55,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:55,435 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:55,435 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:55,435 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:55,435 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:55,435 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,435 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,436 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,436 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:55,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,437 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,437 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:55,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:55,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:55,438 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:55,438 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:55,438 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:55,438 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:55,438 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:55,438 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:55,438 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:55,438 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:55,438 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:55,439 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:55,439 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:55,439 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,439 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:55,439 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:55,440 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,440 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,440 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,440 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:55,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b5bcda0[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:55,442 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:55,442 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,443 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,444 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:55,444 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:55,444 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:55,444 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,444 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:55,449 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,449 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:55,449 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,449 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:55,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:55,451 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:55,451 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:55,451 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:55,452 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:55,452 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:55,453 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,453 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,453 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:55,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:55,454 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,455 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,458 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,458 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:55,458 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,458 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:55,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:55,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:55,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:55,458 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:55,458 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,458 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,458 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:55,458 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:55,459 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:55,458 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,459 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,559 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:55,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:55,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:55,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:55,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:55,562 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:55,562 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:55,562 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:55,562 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:55,563 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,563 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,564 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:55,564 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:55,564 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,564 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:55,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:55,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:55,565 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:55,565 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:55,566 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:55,566 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:55,566 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:55,566 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:55,566 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:55,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:55,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858555566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,569 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,569 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,569 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,569 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:55,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@313e2c74[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:55,570 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:55,570 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:55,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858555569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:55,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858555569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,572 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,572 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:55,572 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:55,572 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,572 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:55,572 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:55,573 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,573 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:55,574 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,574 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:55,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:55,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858555572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:55,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:55,576 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:55,576 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:55,576 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:55,576 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:55,576 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:55,578 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,578 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,578 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:55,578 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,579 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,583 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:55,583 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:55,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:55,583 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,583 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:55,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:55,583 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,583 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,584 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:55,584 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:55,584 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:55,583 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:55,584 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:55,584 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:55,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858555618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,684 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:55,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:55,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:55,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:55,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:55,687 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:55,687 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:55,687 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:55,687 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:55,688 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,688 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,689 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:55,689 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:55,689 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,689 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:55,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:55,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:55,691 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:55,691 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:55,691 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:55,691 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:55,691 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:55,691 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:55,691 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:55,691 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:55,691 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:55,691 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:55,691 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:55,691 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,691 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:55,691 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:55,693 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,693 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,693 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,693 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:55,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@174536a2[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:55,694 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:55,694 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,696 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:55,696 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:55,696 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:55,696 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,696 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:55,697 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/d8fef93ede284e9ebff73606fed07a29 2023-04-18 22:54:55,698 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:55,698 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:55,698 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,698 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:55,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:55,701 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:55,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:55,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:55,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:55,701 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:55,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:55,705 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,706 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,709 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:55,709 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:55,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:55,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:55,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:55,709 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:55,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:55,709 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:55,709 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:55,709 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:55,709 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:55,710 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,710 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,710 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,710 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,710 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,810 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:55,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:55,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:55,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:55,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:55,813 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:55,813 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:55,813 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:55,813 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:55,814 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,814 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,816 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:55,816 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,817 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,817 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:55,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:55,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:55,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:55,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:55,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:55,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:55,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:55,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:55,818 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:55,818 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:55,818 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:55,818 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:55,818 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:55,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:55,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:55,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:55,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,822 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,822 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:55,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16853ff5[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:55,823 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:55,823 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,825 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,825 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:55,825 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:55,825 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:55,825 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,825 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:55,827 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,827 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:55,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,827 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:55,828 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:55,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:55,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:55,830 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:55,830 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:55,830 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:55,830 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:55,830 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:55,835 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,835 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,835 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,837 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,838 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,840 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:55,840 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:55,840 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:55,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:55,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:55,841 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:55,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:55,840 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,841 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,841 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:55,841 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,841 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:55,841 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,841 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,841 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:55,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858555919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,941 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:55,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:55,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:55,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:55,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:55,946 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:55,946 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:55,946 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:55,946 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:55,946 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,946 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,947 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,948 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:55,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:55,948 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,948 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:55,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:55,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:55,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:55,949 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:55,949 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:55,949 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:55,950 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:55,950 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:55,950 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:55,950 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:55,950 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:55,950 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:55,950 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:55,950 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:55,950 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,950 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:55,950 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:55,952 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,952 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,952 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,952 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:55,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6346d648[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:55,954 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:55,954 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,956 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,956 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:55,956 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:55,956 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,956 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:55,956 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:55,957 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:55,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:55,960 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:55,960 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:55,960 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:55,960 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:55,960 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:55,961 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,961 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:55,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,961 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:55,962 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:55,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,965 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,967 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,967 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,967 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:55,967 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:55,967 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:55,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:55,967 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:55,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:55,967 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:55,967 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:55,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:55,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:55,967 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:55,967 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:55,967 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:55,967 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,067 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:56,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:56,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:56,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:56,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:56,071 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:56,071 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:56,071 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:56,071 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:56,071 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,072 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,074 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,074 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:56,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:56,074 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,074 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:56,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:56,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:56,075 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:56,075 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:56,075 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:56,075 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:56,075 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:56,076 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:56,076 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:56,076 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:56,076 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:56,076 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:56,078 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,078 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:56,078 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,078 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:56,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:56,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@235b0bd8[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:56,079 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:56,080 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,081 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,081 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:56,081 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:56,081 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,081 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:56,081 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:56,082 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,082 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:56,082 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:56,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:56,083 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:56,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:56,085 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:56,085 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:56,085 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:56,085 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:56,085 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:56,088 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,088 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,088 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:56,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:56,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,089 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:56,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,090 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,093 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,093 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,093 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,093 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:56,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:56,093 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:56,093 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,093 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:56,093 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:56,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:56,093 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:56,093 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,093 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:56,093 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:56,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:56,109 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/b9b7afea882e4b7080601a6223728058 2023-04-18 22:54:56,194 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:56,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:56,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:56,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:56,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:56,197 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:56,197 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:56,197 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:56,197 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:56,197 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,197 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,199 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,199 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:56,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:56,199 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,199 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:56,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:56,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:56,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:56,200 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:56,200 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:56,200 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:56,200 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:56,200 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:56,200 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:56,200 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:56,200 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:56,200 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:56,201 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:56,201 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:56,201 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,201 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:56,201 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:56,203 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,203 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:56,203 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:56,203 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:56,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:56,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:56,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d15d937[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:56,204 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:56,204 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,207 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:56,207 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,207 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:56,208 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:56,211 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:56,211 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:56,211 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:56,211 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:56,211 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:56,213 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,213 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:56,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,213 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,213 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:56,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,215 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:56,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,220 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,220 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:56,220 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:56,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:56,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:56,221 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,221 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:56,221 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:56,221 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,221 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:56,221 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:56,221 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:56,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:56,221 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,221 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,321 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:56,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:56,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:56,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:56,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:56,325 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:56,325 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:56,325 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:56,325 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:56,325 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,325 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,326 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:56,326 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:56,327 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,327 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:56,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:56,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:56,328 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:56,328 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:56,328 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:56,328 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:56,328 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:56,328 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:56,328 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:56,328 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:56,328 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:56,328 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:56,328 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:56,330 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:56,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:56,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22d027f7[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:56,332 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:56,332 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,333 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,333 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:56,333 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:56,333 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:56,333 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,333 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:56,335 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:56,335 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:56,336 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:56,336 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:56,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:56,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:56,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:56,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:56,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:56,339 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:56,340 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:56,340 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:56,340 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:56,340 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:56,341 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,341 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,342 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:56,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:56,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:56,342 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,343 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:56,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:56,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:56,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:56,346 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:56,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:56,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:56,346 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:56,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,346 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:56,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:56,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:56,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58758 deadline: 1681858556426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,446 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:56,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:56,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:56,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:56,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:56,450 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:56,450 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:56,450 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:56,450 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:56,450 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,450 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,451 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:56,451 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:56,452 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,452 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:56,453 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:56,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:56,453 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:56,453 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:56,453 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:56,453 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:56,453 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:56,453 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:56,454 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:56,454 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:56,454 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:56,454 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:56,454 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:56,454 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,454 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:56,454 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:56,457 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,457 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:56,457 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,457 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:56,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:56,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77714173[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:56,458 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:56,459 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,460 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,460 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:56,460 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,460 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:56,460 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:56,460 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:56,461 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,461 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:56,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:56,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:56,461 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:56,462 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:56,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:56,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:56,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:56,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:56,464 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:56,464 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:56,464 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:56,464 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:56,464 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:56,465 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,465 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:56,465 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:56,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:56,466 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,467 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,470 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,470 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,470 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:56,470 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:56,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:56,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:56,470 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:56,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:56,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:56,470 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,471 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:56,471 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:56,471 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:56,471 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:56,471 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,471 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:56,518 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/9b9b0c28361e4bd29f3feb6f75e2685d 2023-04-18 22:54:56,522 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/d8fef93ede284e9ebff73606fed07a29 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d8fef93ede284e9ebff73606fed07a29 2023-04-18 22:54:56,526 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d8fef93ede284e9ebff73606fed07a29, entries=200, sequenceid=267, filesize=14.3 K 2023-04-18 22:54:56,527 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/b9b7afea882e4b7080601a6223728058 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/b9b7afea882e4b7080601a6223728058 2023-04-18 22:54:56,531 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/b9b7afea882e4b7080601a6223728058, entries=150, sequenceid=267, filesize=11.9 K 2023-04-18 22:54:56,531 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/9b9b0c28361e4bd29f3feb6f75e2685d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/9b9b0c28361e4bd29f3feb6f75e2685d 2023-04-18 22:54:56,534 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/9b9b0c28361e4bd29f3feb6f75e2685d, entries=150, sequenceid=267, filesize=11.9 K 2023-04-18 22:54:56,535 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 94dcab632e7fc6a2d56c5605ce1488fa in 1246ms, sequenceid=267, compaction requested=true 2023-04-18 22:54:56,535 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:56,535 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:56,535 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:56,535 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:56,535 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:56,535 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:56,537 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39176 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:56,537 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/A is initiating minor compaction (all files) 2023-04-18 22:54:56,537 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/A in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:56,537 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/690477a63cc647d695d2898ff398bbe3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/918c6daa773f4f039352864a71874aa4, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d8fef93ede284e9ebff73606fed07a29] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=38.3 K 2023-04-18 22:54:56,537 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36736 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:56,537 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/B is initiating minor compaction (all files) 2023-04-18 22:54:56,537 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/B in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:56,537 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/d22e0e0335d846d6ad3c0fb57044b818, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/83d5b7b2318047618188f488e5ba40df, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/b9b7afea882e4b7080601a6223728058] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=35.9 K 2023-04-18 22:54:56,537 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 690477a63cc647d695d2898ff398bbe3, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681858493131 2023-04-18 22:54:56,538 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting d22e0e0335d846d6ad3c0fb57044b818, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681858493131 2023-04-18 22:54:56,538 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 918c6daa773f4f039352864a71874aa4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1681858493448 2023-04-18 22:54:56,538 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 83d5b7b2318047618188f488e5ba40df, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1681858493448 2023-04-18 22:54:56,538 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting d8fef93ede284e9ebff73606fed07a29, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1681858495167 2023-04-18 22:54:56,538 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting b9b7afea882e4b7080601a6223728058, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1681858495176 2023-04-18 22:54:56,549 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#B#compaction#270 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:56,549 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#A#compaction#269 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:56,558 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/94e5f331477f4f99ba12b50bc1712b41 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/94e5f331477f4f99ba12b50bc1712b41 2023-04-18 22:54:56,561 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/a8d43003feac47dbb45f0d9cdaedd7c4 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/a8d43003feac47dbb45f0d9cdaedd7c4 2023-04-18 22:54:56,563 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/B of 94dcab632e7fc6a2d56c5605ce1488fa into 94e5f331477f4f99ba12b50bc1712b41(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:56,564 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:56,564 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/B, priority=13, startTime=1681858496535; duration=0sec 2023-04-18 22:54:56,564 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:56,564 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:56,565 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36736 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:56,565 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/C is initiating minor compaction (all files) 2023-04-18 22:54:56,565 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/C in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:56,565 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/cbb51ea96ec34ab393007f8bbc3ea413, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/935dd618655d4d74862b8195b3b4ef9d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/9b9b0c28361e4bd29f3feb6f75e2685d] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=35.9 K 2023-04-18 22:54:56,566 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting cbb51ea96ec34ab393007f8bbc3ea413, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681858493131 2023-04-18 22:54:56,566 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 935dd618655d4d74862b8195b3b4ef9d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1681858493448 2023-04-18 22:54:56,566 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/A of 94dcab632e7fc6a2d56c5605ce1488fa into a8d43003feac47dbb45f0d9cdaedd7c4(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:56,566 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:56,566 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/A, priority=13, startTime=1681858496535; duration=0sec 2023-04-18 22:54:56,566 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 9b9b0c28361e4bd29f3feb6f75e2685d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1681858495176 2023-04-18 22:54:56,566 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:56,571 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:56,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:56,572 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#C#compaction#271 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:56,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:56,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:56,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:56,576 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:56,576 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:56,576 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:56,576 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:56,576 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,576 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,578 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,578 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:56,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:56,578 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:56,578 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:56,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:56,579 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:56,579 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:56,579 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:56,579 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:56,580 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:56,580 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:56,580 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:56,580 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:54:56,580 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:54:56,580 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:56,580 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:54:56,580 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:56,580 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:54:56,580 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:56,581 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/e9540b06c2cf4deeb8a5a620e69eb4c1 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/e9540b06c2cf4deeb8a5a620e69eb4c1 2023-04-18 22:54:56,585 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/C of 94dcab632e7fc6a2d56c5605ce1488fa into e9540b06c2cf4deeb8a5a620e69eb4c1(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:56,585 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:56,585 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/C, priority=13, startTime=1681858496535; duration=0sec 2023-04-18 22:54:56,585 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:56,990 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/a9e91e8c25e749fdb90b0058f2542925 2023-04-18 22:54:57,404 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/733397e581204e9dad6baea0fa2ab58b 2023-04-18 22:54:57,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:57,439 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:57,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858497473, max timestamp=1681858497485Num uniques 859; 2023-04-18 22:54:57,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:54:57,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858497473, max timestamp=1681858497485Num uniques 859; 2023-04-18 22:54:57,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858497473, max timestamp=1681858497485Num uniques 859; 2023-04-18 22:54:57,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:54:57,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:54:57,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:57,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858557533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:57,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858557572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,575 DEBUG [Thread-1754] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4126 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:54:57,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:57,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858557586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,588 DEBUG [Thread-1760] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:54:57,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:57,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858557590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,592 DEBUG [Thread-1756] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:54:57,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:57,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858557594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,596 DEBUG [Thread-1758] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:54:57,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:57,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858557636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,815 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/dacb2dbd785249ff9b7c8abd6b86d068 2023-04-18 22:54:57,820 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/a9e91e8c25e749fdb90b0058f2542925 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/a9e91e8c25e749fdb90b0058f2542925 2023-04-18 22:54:57,823 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/a9e91e8c25e749fdb90b0058f2542925, entries=150, sequenceid=279, filesize=11.9 K 2023-04-18 22:54:57,824 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/733397e581204e9dad6baea0fa2ab58b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/733397e581204e9dad6baea0fa2ab58b 2023-04-18 22:54:57,827 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/733397e581204e9dad6baea0fa2ab58b, entries=150, sequenceid=279, filesize=11.9 K 2023-04-18 22:54:57,828 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/dacb2dbd785249ff9b7c8abd6b86d068 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/dacb2dbd785249ff9b7c8abd6b86d068 2023-04-18 22:54:57,832 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/dacb2dbd785249ff9b7c8abd6b86d068, entries=150, sequenceid=279, filesize=11.9 K 2023-04-18 22:54:57,832 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 94dcab632e7fc6a2d56c5605ce1488fa in 1252ms, sequenceid=279, compaction requested=false 2023-04-18 22:54:57,832 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:57,832 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:57,834 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,834 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:57,834 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:57,835 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:57,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:57,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c8f818e[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:57,836 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:57,836 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:57,837 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:57,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:57,837 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:57,838 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,838 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:57,840 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:57,840 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,840 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:57,840 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:57,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:57,841 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-18 22:54:57,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:54:57,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:54:57,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:54:57,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:54:57,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:54:57,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:54:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:57,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:57,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:57,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:57,845 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:57,845 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:57,845 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:57,845 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:57,845 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:57,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:57,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:57,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:57,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:57,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:57,847 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,849 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,851 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,851 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:57,851 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:57,851 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:57,851 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:57,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:57,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:57,851 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,851 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:57,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:57,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:57,851 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:57,852 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:57,852 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:57,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:57,852 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:57,852 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:57,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:57,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858557861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,952 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:57,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:57,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:57,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:57,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:57,955 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:57,955 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:57,955 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:57,955 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:57,956 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:57,956 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:57,961 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,961 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:57,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:57,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:57,962 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,962 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:57,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:57,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:57,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:57,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:57,963 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:57,963 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:57,963 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:57,963 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:57,963 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:57,964 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:57,964 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:57,964 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:57,964 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:57,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:57,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858557964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,965 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,965 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:57,966 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:57,966 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:57,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:57,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f602d49[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:57,967 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:57,967 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:57,968 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:57,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:57,969 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:57,969 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:57,969 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,969 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:57,969 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:57,970 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:57,970 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:57,970 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:57,970 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:57,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:57,972 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:57,973 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:57,973 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:57,973 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:57,973 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:57,975 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:57,975 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:57,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:57,976 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:57,976 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:57,978 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,980 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,980 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:57,980 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:57,980 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:57,980 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:57,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:57,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:57,980 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:57,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:57,980 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:57,980 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:57,980 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:57,980 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:57,980 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:57,980 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:57,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:57,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:58,080 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:58,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:58,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:58,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:58,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:58,084 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:58,084 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:58,084 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:58,084 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:58,084 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,084 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,086 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:58,086 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:58,086 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,086 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:58,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:58,087 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:58,087 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:58,087 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:58,087 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:58,087 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:58,087 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:58,087 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:58,088 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:58,088 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:58,088 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:58,088 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:58,088 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,088 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:58,088 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:58,089 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,089 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,089 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,089 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:58,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c2c4acd[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:58,091 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:58,091 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,093 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,094 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:58,094 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:58,094 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:58,094 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,094 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:58,095 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,095 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:58,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,095 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:58,095 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:58,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:58,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:58,097 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:58,097 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:58,097 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:58,097 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:58,097 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:58,099 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,099 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,099 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:58,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,099 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,101 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,103 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,103 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:58,103 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:58,103 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:58,103 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:58,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:58,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:58,103 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,103 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:58,103 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:58,103 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:58,103 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,104 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,104 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:58,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858558167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,204 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:58,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:58,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:58,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:58,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:58,207 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:58,207 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:58,207 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:58,207 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:58,208 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,208 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,210 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,210 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:58,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:58,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:58,210 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,210 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:58,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:58,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:58,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:58,211 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:58,211 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:58,211 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:58,211 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:58,211 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:58,212 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:58,212 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:58,212 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:58,212 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:58,212 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:58,212 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:58,212 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,212 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:58,212 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:58,213 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,213 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,214 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,214 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:58,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61abeae2[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:58,215 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:58,215 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,216 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,216 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:58,216 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:58,216 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:58,216 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,216 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:58,218 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,218 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:58,218 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,218 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:58,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:58,220 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:58,220 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:58,220 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:58,220 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:58,220 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:58,222 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,222 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,222 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:58,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,223 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,224 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,226 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,226 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,226 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,226 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:58,226 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:58,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:58,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:58,226 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:58,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:58,226 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:58,226 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:58,226 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:58,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:58,226 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,227 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,227 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:58,251 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/88a0763066ef4bcf84099cbe32e64a51 2023-04-18 22:54:58,327 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:58,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:58,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:58,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:58,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:58,330 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:58,330 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:58,330 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:58,330 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:58,330 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,330 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,332 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:58,332 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:58,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:58,332 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,332 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:58,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:58,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:58,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:58,334 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:58,334 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:58,334 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:58,334 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:58,339 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,339 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,340 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,340 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:58,342 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:58,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11d884a8[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:58,342 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,343 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,343 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:58,343 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:58,343 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:58,343 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,344 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:58,345 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:58,345 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:58,345 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,345 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:58,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:58,348 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:58,348 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:58,348 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:58,348 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:58,348 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:58,350 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,350 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,350 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:58,351 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,352 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,355 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,355 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:58,355 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:58,355 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:58,356 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:58,356 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:58,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:58,356 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,356 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,356 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,356 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:58,356 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:58,356 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:58,456 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:58,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:58,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:58,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:58,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:58,460 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:58,460 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:58,460 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:58,460 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:58,460 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,460 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,462 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,462 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:58,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:58,462 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,462 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:58,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:58,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:58,463 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:58,463 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:58,463 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:58,463 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:58,464 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:58,464 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:58,464 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:58,464 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:58,464 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:58,464 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:58,464 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:58,464 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,464 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:58,464 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:58,465 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,465 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,466 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,466 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:58,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f6ce9bf[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:58,467 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:58,467 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,469 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,470 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:58,470 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:58,470 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:58,470 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,470 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:58,471 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:58,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858558469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:58,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:58,474 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:58,474 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:58,474 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:58,474 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:58,474 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:58,476 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,476 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:58,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,476 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:58,476 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,478 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,479 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:58,480 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:58,480 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,480 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,480 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:58,480 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,480 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:58,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:58,480 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,480 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:58,480 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,480 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:58,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:58,480 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:58,581 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:58,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:58,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:58,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:58,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:58,584 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:58,584 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:58,584 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:58,584 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:58,584 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,584 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,586 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:58,586 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,587 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,587 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:58,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:58,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:58,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:58,588 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:58,588 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:58,588 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:58,588 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:58,589 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:58,590 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,591 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,591 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:58,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5168a4bd[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:58,592 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:58,592 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,593 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,593 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:58,594 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:58,594 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:58,594 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,594 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:58,595 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,595 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:58,595 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:58,595 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:58,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:58,598 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:58,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:58,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:58,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:58,599 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:58,601 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,601 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,601 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:58,602 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,603 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,605 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:58,605 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:58,605 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:58,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:58,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:58,605 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:58,605 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,606 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,606 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,606 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,606 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,606 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:58,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:58,606 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:58,606 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,663 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/a9991a3c4e544f28999e80a3d86172bd 2023-04-18 22:54:58,706 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:58,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:58,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:58,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:58,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:58,709 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:58,710 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:58,710 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:58,710 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:58,710 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,710 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,712 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:58,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:58,712 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:58,713 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,713 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:58,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:58,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:58,714 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:58,714 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:58,714 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:58,714 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:58,714 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:58,714 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:58,714 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:58,714 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:58,714 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:58,714 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:58,714 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:58,714 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,714 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:58,714 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,716 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:58,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f2ab888[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:58,718 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:58,718 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,725 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,725 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:58,725 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:58,725 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:58,725 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,726 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:58,729 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:58,729 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,729 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,729 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:58,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:58,732 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:58,732 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:58,732 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:58,732 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:58,732 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:58,733 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,733 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,733 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,734 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,735 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:58,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:58,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:58,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:58,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,738 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,738 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,738 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:58,737 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:58,738 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:58,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:58,738 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:58,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:58,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:58,838 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:58,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:58,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:58,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:58,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:58,844 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:58,844 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:58,844 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:58,844 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:58,845 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,845 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,847 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,847 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:58,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:58,848 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,848 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:58,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:58,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:58,849 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:58,849 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:58,849 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:58,849 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:58,851 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,851 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,851 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,851 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:58,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a028c01[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:58,853 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:58,853 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,854 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,854 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:58,854 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:58,854 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:58,854 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,854 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:58,856 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:58,856 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:58,856 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,856 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:58,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:58,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:58,858 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:58,858 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:58,859 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:58,859 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:58,859 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:58,864 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,864 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,864 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:58,865 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,866 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:58,868 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:58,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:54:58,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:58,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:58,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:58,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:58,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,869 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,869 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,869 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,869 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,869 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,871 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:58,871 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:58,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:58,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:58,971 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:58,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:58,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:58,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:58,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:58,975 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:58,975 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:58,975 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:58,975 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:58,975 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,975 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:54:58,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858558973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,976 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,976 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:58,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:58,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:58,977 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,977 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:58,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:58,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:58,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:58,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:58,978 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:58,978 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:58,978 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:58,978 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:58,978 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:58,978 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:58,978 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:58,978 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:54:58,978 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:58,978 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:54:58,978 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:54:58,978 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,978 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:54:58,979 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:54:58,980 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,980 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,980 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,980 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:54:58,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:54:58,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72ebabef[Count = 0] remaining members to acquire global barrier 2023-04-18 22:54:58,982 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:54:58,982 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,983 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,984 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:54:58,984 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:54:58,984 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,984 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:54:58,984 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:54:58,985 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,985 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:54:58,985 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:58,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,985 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:54:58,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:54:58,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:54:58,988 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:54:58,988 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:54:58,988 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:54:58,988 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:54:58,988 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:54:58,990 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,990 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:54:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:54:58,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,990 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:54:58,991 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:54:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,992 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:54:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:54:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,994 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,994 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,994 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:58,994 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:54:58,994 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:58,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:58,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:58,994 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:58,994 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,995 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:54:58,995 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:58,994 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:54:58,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:54:58,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:54:58,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:54:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:54:58,995 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:54:58,995 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:54:59,076 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/d9aa2c8c96144d8c9ea8866ec6388cb9 2023-04-18 22:54:59,080 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/88a0763066ef4bcf84099cbe32e64a51 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/88a0763066ef4bcf84099cbe32e64a51 2023-04-18 22:54:59,084 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/88a0763066ef4bcf84099cbe32e64a51, entries=150, sequenceid=318, filesize=11.9 K 2023-04-18 22:54:59,084 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/a9991a3c4e544f28999e80a3d86172bd as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/a9991a3c4e544f28999e80a3d86172bd 2023-04-18 22:54:59,087 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/a9991a3c4e544f28999e80a3d86172bd, entries=150, sequenceid=318, filesize=11.9 K 2023-04-18 22:54:59,088 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/d9aa2c8c96144d8c9ea8866ec6388cb9 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/d9aa2c8c96144d8c9ea8866ec6388cb9 2023-04-18 22:54:59,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/d9aa2c8c96144d8c9ea8866ec6388cb9, entries=150, sequenceid=318, filesize=11.9 K 2023-04-18 22:54:59,092 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 94dcab632e7fc6a2d56c5605ce1488fa in 1251ms, sequenceid=318, compaction requested=true 2023-04-18 22:54:59,092 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:59,092 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:59,092 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:59,092 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:59,092 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:59,092 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:59,095 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:59,095 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/B is initiating minor compaction (all files) 2023-04-18 22:54:59,095 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:59,095 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/B in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:59,095 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/A is initiating minor compaction (all files) 2023-04-18 22:54:59,095 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:54:59,095 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/A in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:59,095 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/a8d43003feac47dbb45f0d9cdaedd7c4, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/a9e91e8c25e749fdb90b0058f2542925, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/88a0763066ef4bcf84099cbe32e64a51] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=36.3 K 2023-04-18 22:54:59,095 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/94e5f331477f4f99ba12b50bc1712b41, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/733397e581204e9dad6baea0fa2ab58b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/a9991a3c4e544f28999e80a3d86172bd] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=36.3 K 2023-04-18 22:54:59,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:54:59,096 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting a8d43003feac47dbb45f0d9cdaedd7c4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1681858495176 2023-04-18 22:54:59,096 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting a9e91e8c25e749fdb90b0058f2542925, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681858495291 2023-04-18 22:54:59,096 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 94e5f331477f4f99ba12b50bc1712b41, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1681858495176 2023-04-18 22:54:59,097 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 88a0763066ef4bcf84099cbe32e64a51, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681858497516 2023-04-18 22:54:59,097 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 733397e581204e9dad6baea0fa2ab58b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681858495291 2023-04-18 22:54:59,097 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting a9991a3c4e544f28999e80a3d86172bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681858497516 2023-04-18 22:54:59,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:54:59,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:54:59,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:54:59,101 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:54:59,101 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:59,101 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:54:59,101 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:54:59,101 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:59,101 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:59,103 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#A#compaction#278 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:59,105 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#B#compaction#279 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:59,109 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:59,109 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:54:59,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:54:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:54:59,109 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:54:59,109 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:54:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:59,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:54:59,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:54:59,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:54:59,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:54:59,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:54:59,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:54:59,110 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:54:59,110 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:54:59,111 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:54:59,111 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:54:59,111 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:54:59,111 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:59,111 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:54:59,111 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-18 22:54:59,111 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:54:59,111 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:59,111 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:54:59,111 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:59,111 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:54:59,112 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:54:59,514 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/580eed59975c4882a675bbacadeebd2e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/580eed59975c4882a675bbacadeebd2e 2023-04-18 22:54:59,515 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/c79c933be7bc4b69a623cdbc467fcacf as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/c79c933be7bc4b69a623cdbc467fcacf 2023-04-18 22:54:59,519 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/B of 94dcab632e7fc6a2d56c5605ce1488fa into 580eed59975c4882a675bbacadeebd2e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:59,519 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:59,519 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/B, priority=13, startTime=1681858499092; duration=0sec 2023-04-18 22:54:59,519 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:54:59,519 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:54:59,520 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/d4d6b3870a0a414491000c7cac70577f 2023-04-18 22:54:59,521 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/A of 94dcab632e7fc6a2d56c5605ce1488fa into c79c933be7bc4b69a623cdbc467fcacf(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:59,521 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:59,521 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/A, priority=13, startTime=1681858499092; duration=0sec 2023-04-18 22:54:59,521 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:59,522 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:54:59,522 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/C is initiating minor compaction (all files) 2023-04-18 22:54:59,523 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/C in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:54:59,523 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/e9540b06c2cf4deeb8a5a620e69eb4c1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/dacb2dbd785249ff9b7c8abd6b86d068, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/d9aa2c8c96144d8c9ea8866ec6388cb9] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=36.3 K 2023-04-18 22:54:59,523 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting e9540b06c2cf4deeb8a5a620e69eb4c1, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1681858495176 2023-04-18 22:54:59,524 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting dacb2dbd785249ff9b7c8abd6b86d068, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1681858495291 2023-04-18 22:54:59,524 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting d9aa2c8c96144d8c9ea8866ec6388cb9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681858497516 2023-04-18 22:54:59,542 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#C#compaction#282 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:54:59,554 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/a4511808f017450aacebcf96663bb411 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/a4511808f017450aacebcf96663bb411 2023-04-18 22:54:59,559 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/C of 94dcab632e7fc6a2d56c5605ce1488fa into a4511808f017450aacebcf96663bb411(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:54:59,559 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:54:59,559 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/C, priority=13, startTime=1681858499092; duration=0sec 2023-04-18 22:54:59,559 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:54:59,938 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/d1a31c7da85f4d819346e70dd08530e0 2023-04-18 22:54:59,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:54:59,983 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:55:00,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858500002, max timestamp=1681858500020Num uniques 859; 2023-04-18 22:55:00,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:55:00,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858500002, max timestamp=1681858500020Num uniques 859; 2023-04-18 22:55:00,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858500002, max timestamp=1681858500020Num uniques 859; 2023-04-18 22:55:00,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:55:00,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:55:00,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858500045, max timestamp=1681858500075Num uniques 1018; 2023-04-18 22:55:00,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-18 22:55:00,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858500045, max timestamp=1681858500075Num uniques 1018; 2023-04-18 22:55:00,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858500045, max timestamp=1681858500075Num uniques 1018; 2023-04-18 22:55:00,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-18 22:55:00,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-18 22:55:00,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681858500113, max timestamp=1681858500133Num uniques 1027; 2023-04-18 22:55:00,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 22:55:00,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681858500113, max timestamp=1681858500133Num uniques 1027; 2023-04-18 22:55:00,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 22:55:00,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681858500113, max timestamp=1681858500133Num uniques 1027; 2023-04-18 22:55:00,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 22:55:00,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681858500176, max timestamp=1681858500188Num uniques 1036; 2023-04-18 22:55:00,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 22:55:00,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681858500176, max timestamp=1681858500188Num uniques 1036; 2023-04-18 22:55:00,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 22:55:00,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681858500176, max timestamp=1681858500188Num uniques 1036; 2023-04-18 22:55:00,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 22:55:00,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681858500233, max timestamp=1681858500243Num uniques 1045; 2023-04-18 22:55:00,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-18 22:55:00,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681858500233, max timestamp=1681858500243Num uniques 1045; 2023-04-18 22:55:00,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-18 22:55:00,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681858500233, max timestamp=1681858500243Num uniques 1045; 2023-04-18 22:55:00,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-18 22:55:00,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:00,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858560306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,363 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/8c2a9c092d5f455380af8421f95b1657 2023-04-18 22:55:00,368 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/d4d6b3870a0a414491000c7cac70577f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d4d6b3870a0a414491000c7cac70577f 2023-04-18 22:55:00,371 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d4d6b3870a0a414491000c7cac70577f, entries=150, sequenceid=327, filesize=11.9 K 2023-04-18 22:55:00,372 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/d1a31c7da85f4d819346e70dd08530e0 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/d1a31c7da85f4d819346e70dd08530e0 2023-04-18 22:55:00,376 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/d1a31c7da85f4d819346e70dd08530e0, entries=150, sequenceid=327, filesize=11.9 K 2023-04-18 22:55:00,377 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/8c2a9c092d5f455380af8421f95b1657 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/8c2a9c092d5f455380af8421f95b1657 2023-04-18 22:55:00,386 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/8c2a9c092d5f455380af8421f95b1657, entries=150, sequenceid=327, filesize=11.9 K 2023-04-18 22:55:00,387 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 94dcab632e7fc6a2d56c5605ce1488fa in 1276ms, sequenceid=327, compaction requested=false 2023-04-18 22:55:00,387 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:00,387 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:00,387 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:00,388 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:00,388 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,391 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:00,391 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:00,392 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,392 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:00,393 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,393 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:00,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:00,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6632a443[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:00,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:00,396 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,397 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,397 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:00,397 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:00,397 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,397 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:00,397 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:00,399 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:00,399 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:00,399 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:00,399 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:00,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:00,401 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:00,401 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:00,402 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:00,402 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:00,402 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:00,407 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,407 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:00,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,407 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:00,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:00,408 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:00,409 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:55:00,411 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=147.60 KB heapSize=422.16 KB 2023-04-18 22:55:00,411 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:55:00,411 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:55:00,411 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:55:00,411 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:55:00,412 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:55:00,412 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:55:00,414 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:00,414 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:00,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:00,415 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,415 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,415 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,415 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,415 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,415 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,415 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,415 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:00,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:00,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:00,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:00,415 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:00,415 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:00,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:00,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:00,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858560428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,515 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:00,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:00,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:00,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:00,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:00,519 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:00,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:00,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:00,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:00,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,520 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:00,520 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:00,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:00,521 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,521 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:00,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:00,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:00,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:00,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:00,522 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:00,522 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:00,522 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:00,522 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:00,522 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:00,522 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:00,522 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:00,522 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:55:00,522 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:00,523 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:00,523 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:00,523 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,523 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:00,523 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:00,528 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,528 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:00,529 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,529 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:00,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:00,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ce6aef5[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:00,530 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:00,531 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,532 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,532 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:00,532 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,532 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:00,532 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:00,532 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:00,533 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:00,533 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:00,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:00,533 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:00,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858560532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,533 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:00,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:00,536 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:00,536 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:00,536 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:00,536 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:00,536 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:00,538 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,538 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:00,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,538 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:00,538 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,540 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,543 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,543 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,543 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,543 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:00,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:00,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:00,543 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,543 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:00,543 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:00,543 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,543 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:00,543 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:00,543 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:00,543 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:00,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:00,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:00,643 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:00,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:00,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:00,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:00,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:00,649 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:00,649 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:00,649 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:00,649 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:00,649 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,649 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,650 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:00,650 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:00,651 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,651 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:00,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:00,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:00,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:00,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:00,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:00,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:00,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:00,652 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:00,652 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:00,652 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:55:00,652 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:00,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:00,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:00,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:00,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:00,654 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,654 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:00,654 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,654 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:00,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:00,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:00,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:00,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@722a53ec[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:00,656 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:00,656 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,657 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,657 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:00,657 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:00,657 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,657 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:00,657 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:00,659 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,659 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:00,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:00,659 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:00,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:00,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:00,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:00,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:00,662 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:00,662 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:00,662 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:00,662 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:00,662 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:00,664 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,664 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:00,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,664 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:00,664 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:00,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:00,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:00,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:00,665 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,668 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,668 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,668 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:00,668 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:00,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:00,668 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:00,668 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,668 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:00,668 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:00,668 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:00,668 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,668 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:00,668 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:00,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:00,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:00,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858560734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,768 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:00,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:00,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:00,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:00,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:00,771 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:00,772 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:00,772 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:00,772 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:00,772 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,772 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,774 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:00,774 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:00,774 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,774 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:00,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:00,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:00,775 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:00,775 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:00,775 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:00,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:00,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:00,776 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:00,776 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:00,776 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:55:00,776 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:00,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:00,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:00,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:00,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:00,783 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,783 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5db13a95[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:00,785 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:00,786 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,787 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,787 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:00,787 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:00,787 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:00,787 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,787 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:00,788 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,788 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:00,788 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:00,788 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:00,791 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:00,791 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:00,791 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:00,791 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:00,791 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:00,792 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,792 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:00,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,793 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:00,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:00,793 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:00,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,794 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,800 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:00,800 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:00,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:00,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:00,801 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:00,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:00,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:00,801 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,801 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,801 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,801 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,801 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,801 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:00,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:00,801 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:00,801 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:00,821 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/ca6151c3d4714a988b1ed18da5782a2d 2023-04-18 22:55:00,901 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:00,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:00,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:00,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:00,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:00,905 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:00,905 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:00,905 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:00,905 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:00,905 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,905 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,907 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,907 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:00,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:00,907 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,907 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:00,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:00,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:00,908 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:00,909 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:00,909 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:55:00,909 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:00,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:00,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:00,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:00,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:00,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,910 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:00,911 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,911 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:00,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:00,912 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:00,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ed29ddc[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:00,912 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,915 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,915 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:00,915 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:00,915 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:00,915 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,915 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:00,916 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:00,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:00,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:00,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:00,918 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:00,918 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:00,918 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:00,919 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:00,919 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:00,920 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,920 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:00,920 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:00,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:00,921 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,922 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,925 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:00,925 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:00,925 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,925 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:00,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:00,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:00,925 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:00,925 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:00,925 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,925 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:00,925 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:00,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:00,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:00,925 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:00,925 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:00,926 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:01,026 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:01,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:01,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:01,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:01,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:01,029 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:01,029 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:01,029 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:01,029 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:01,030 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,030 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,032 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:01,032 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:01,032 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,032 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:01,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:01,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:01,034 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:01,034 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:01,034 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:01,034 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:01,034 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:01,034 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:01,034 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:01,034 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:55:01,034 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:01,035 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:01,035 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:01,035 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,035 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:01,035 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:01,037 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,037 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,037 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,037 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:01,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47aa0ff5[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:01,039 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:01,039 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,041 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:01,041 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,041 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:01,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858561038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:01,043 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:01,043 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,043 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,043 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:01,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:01,045 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:01,045 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:01,045 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:01,045 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:01,045 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:01,048 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,048 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,048 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:01,049 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,050 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,052 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,052 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:01,052 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:01,052 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:01,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:01,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:01,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:01,052 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:01,053 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:01,052 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,053 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:01,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:01,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:01,053 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,053 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,053 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,053 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,153 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:01,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:01,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:01,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:01,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:01,156 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:01,156 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:01,156 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:01,157 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:01,157 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,157 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,159 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:01,159 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:01,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:01,159 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,159 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:01,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:01,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:01,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:01,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:01,160 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:01,161 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:01,161 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:01,161 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:01,161 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:01,161 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:01,161 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:01,161 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:55:01,161 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:01,161 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:01,161 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:01,161 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,161 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:01,161 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:01,164 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,164 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,164 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,164 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:01,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:01,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@673f91df[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:01,165 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:01,165 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,167 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,168 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:01,168 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:01,168 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:01,168 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,168 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:01,170 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:01,170 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,170 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:01,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,170 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:01,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:01,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:01,172 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:01,172 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:01,172 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:01,172 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:01,172 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:01,175 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,175 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,176 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:01,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:01,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:01,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,176 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,178 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,187 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,187 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:01,187 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:01,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:01,187 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:01,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:01,187 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:01,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:01,187 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,188 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,188 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,188 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,188 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,188 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:01,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:01,188 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:01,232 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/fff35dabfefb4ffbad9d3549cd3906a2 2023-04-18 22:55:01,246 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/83f6a1842bdc46f597db55acbafed72c 2023-04-18 22:55:01,250 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/ca6151c3d4714a988b1ed18da5782a2d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/ca6151c3d4714a988b1ed18da5782a2d 2023-04-18 22:55:01,253 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/ca6151c3d4714a988b1ed18da5782a2d, entries=200, sequenceid=437, filesize=14.3 K 2023-04-18 22:55:01,254 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/fff35dabfefb4ffbad9d3549cd3906a2 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/fff35dabfefb4ffbad9d3549cd3906a2 2023-04-18 22:55:01,257 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/fff35dabfefb4ffbad9d3549cd3906a2, entries=150, sequenceid=437, filesize=11.9 K 2023-04-18 22:55:01,258 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/83f6a1842bdc46f597db55acbafed72c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/83f6a1842bdc46f597db55acbafed72c 2023-04-18 22:55:01,261 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/83f6a1842bdc46f597db55acbafed72c, entries=150, sequenceid=437, filesize=11.9 K 2023-04-18 22:55:01,262 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~422.11 KB/432240, currentSize=33.54 KB/34350 for 94dcab632e7fc6a2d56c5605ce1488fa in 851ms, sequenceid=437, compaction requested=true 2023-04-18 22:55:01,262 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:01,262 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:01,262 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:01,262 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:01,262 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:01,262 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:01,263 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39680 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:01,263 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:01,263 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/A is initiating minor compaction (all files) 2023-04-18 22:55:01,263 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/B is initiating minor compaction (all files) 2023-04-18 22:55:01,264 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/A in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:01,264 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/B in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:01,264 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/c79c933be7bc4b69a623cdbc467fcacf, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d4d6b3870a0a414491000c7cac70577f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/ca6151c3d4714a988b1ed18da5782a2d] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=38.8 K 2023-04-18 22:55:01,264 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/580eed59975c4882a675bbacadeebd2e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/d1a31c7da85f4d819346e70dd08530e0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/fff35dabfefb4ffbad9d3549cd3906a2] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=36.4 K 2023-04-18 22:55:01,264 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting c79c933be7bc4b69a623cdbc467fcacf, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681858497516 2023-04-18 22:55:01,264 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 580eed59975c4882a675bbacadeebd2e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681858497516 2023-04-18 22:55:01,264 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting d4d6b3870a0a414491000c7cac70577f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1681858497853 2023-04-18 22:55:01,265 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting d1a31c7da85f4d819346e70dd08530e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1681858497853 2023-04-18 22:55:01,265 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting ca6151c3d4714a988b1ed18da5782a2d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1681858500295 2023-04-18 22:55:01,265 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting fff35dabfefb4ffbad9d3549cd3906a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1681858500295 2023-04-18 22:55:01,271 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#A#compaction#287 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:01,272 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#B#compaction#288 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:01,288 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:01,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:01,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:01,293 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/882fb29e927045a2835fa72273f5740a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/882fb29e927045a2835fa72273f5740a 2023-04-18 22:55:01,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:01,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:01,295 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:01,295 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:01,295 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:01,295 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:01,296 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,296 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,297 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/8db234c322b24c6db8c6d5b5ff0b6e15 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8db234c322b24c6db8c6d5b5ff0b6e15 2023-04-18 22:55:01,298 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:01,298 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,299 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,299 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:01,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:01,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:01,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:01,300 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:01,300 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:01,300 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:01,300 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:01,300 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:01,300 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:01,300 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:01,300 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/B of 94dcab632e7fc6a2d56c5605ce1488fa into 882fb29e927045a2835fa72273f5740a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:01,300 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-18 22:55:01,300 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:01,301 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/B, priority=13, startTime=1681858501262; duration=0sec 2023-04-18 22:55:01,301 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:55:01,301 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:01,301 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:01,301 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:01,301 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:55:01,301 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:01,301 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:55:01,302 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:01,302 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:01,302 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/C is initiating minor compaction (all files) 2023-04-18 22:55:01,302 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/C in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:01,302 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/a4511808f017450aacebcf96663bb411, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/8c2a9c092d5f455380af8421f95b1657, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/83f6a1842bdc46f597db55acbafed72c] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=36.4 K 2023-04-18 22:55:01,302 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/A of 94dcab632e7fc6a2d56c5605ce1488fa into 8db234c322b24c6db8c6d5b5ff0b6e15(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:01,302 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:01,302 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/A, priority=13, startTime=1681858501262; duration=0sec 2023-04-18 22:55:01,303 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting a4511808f017450aacebcf96663bb411, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681858497516 2023-04-18 22:55:01,303 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:01,303 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 8c2a9c092d5f455380af8421f95b1657, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1681858497853 2023-04-18 22:55:01,303 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 83f6a1842bdc46f597db55acbafed72c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1681858500295 2023-04-18 22:55:01,309 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/d1abdc83f2cf4b1b8c14ddd0fb7bcf83 2023-04-18 22:55:01,310 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#C#compaction#290 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:01,328 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/1f8d71a9b8004c16877e7eed9a28f00d 2023-04-18 22:55:01,332 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/3ce970096b264f9f95c586b558c51af8 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/3ce970096b264f9f95c586b558c51af8 2023-04-18 22:55:01,337 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/C of 94dcab632e7fc6a2d56c5605ce1488fa into 3ce970096b264f9f95c586b558c51af8(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:01,337 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:01,337 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/C, priority=13, startTime=1681858501262; duration=0sec 2023-04-18 22:55:01,337 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:01,348 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/5ead824a7dc54232bb772d0361a74657 2023-04-18 22:55:01,352 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/d1abdc83f2cf4b1b8c14ddd0fb7bcf83 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d1abdc83f2cf4b1b8c14ddd0fb7bcf83 2023-04-18 22:55:01,356 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d1abdc83f2cf4b1b8c14ddd0fb7bcf83, entries=150, sequenceid=446, filesize=11.9 K 2023-04-18 22:55:01,357 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/1f8d71a9b8004c16877e7eed9a28f00d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/1f8d71a9b8004c16877e7eed9a28f00d 2023-04-18 22:55:01,360 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/1f8d71a9b8004c16877e7eed9a28f00d, entries=150, sequenceid=446, filesize=11.9 K 2023-04-18 22:55:01,361 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/5ead824a7dc54232bb772d0361a74657 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/5ead824a7dc54232bb772d0361a74657 2023-04-18 22:55:01,365 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/5ead824a7dc54232bb772d0361a74657, entries=150, sequenceid=446, filesize=11.9 K 2023-04-18 22:55:01,365 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 94dcab632e7fc6a2d56c5605ce1488fa in 65ms, sequenceid=446, compaction requested=false 2023-04-18 22:55:01,365 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:01,365 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:01,366 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:01,366 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:01,366 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,366 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:01,366 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:01,367 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,367 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,368 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,368 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@338e7a3f[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:01,369 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:01,370 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,371 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,371 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:01,372 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,372 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:01,375 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,375 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:01,375 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:01,375 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:01,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:01,379 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:01,379 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:01,379 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:01,379 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:01,379 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:01,381 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,381 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,381 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:01,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,381 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,383 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,385 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,385 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,385 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:01,385 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,386 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:01,386 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,386 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,386 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,386 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:01,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:01,386 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:01,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:01,386 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:01,386 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:01,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:01,486 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:01,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:01,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:01,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:01,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:01,490 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:01,490 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:01,490 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:01,490 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:01,491 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,491 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,493 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,493 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:01,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:01,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:01,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:01,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:01,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:01,495 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:01,495 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:01,495 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:01,495 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:01,495 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:01,495 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:01,495 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:01,496 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:01,496 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:01,496 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:01,496 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:01,496 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,496 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:01,496 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:01,497 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,497 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,498 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,498 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:01,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9083def[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:01,500 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:01,500 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,501 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,501 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:01,501 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:01,501 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:01,501 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,501 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:01,502 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,503 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:01,503 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:01,503 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:01,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:01,505 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:01,505 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:01,505 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:01,505 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:01,505 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:01,508 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,508 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,508 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:01,509 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,510 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,512 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,512 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:01,512 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:01,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:01,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:01,512 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,513 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,513 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,513 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,513 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,513 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:01,512 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:01,513 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:01,513 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:01,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:01,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:55:01,555 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:55:01,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:55:01,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:01,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:55:01,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:01,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:55:01,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:01,578 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/64ea5428b48a4bd98c6b98330b61e344 2023-04-18 22:55:01,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858501592, max timestamp=1681858501604Num uniques 859; 2023-04-18 22:55:01,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:55:01,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858501592, max timestamp=1681858501604Num uniques 859; 2023-04-18 22:55:01,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858501592, max timestamp=1681858501604Num uniques 859; 2023-04-18 22:55:01,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:55:01,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:55:01,613 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:01,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:01,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:01,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:01,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:01,616 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:01,616 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:01,616 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:01,616 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:01,616 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,616 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,618 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,618 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:01,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:01,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:01,618 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,618 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:01,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:01,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:01,619 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:01,619 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:01,619 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:55:01,619 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:01,621 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,621 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,622 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,622 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:01,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:01,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e4a5e9c[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:01,623 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:01,623 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,625 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,625 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:01,625 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:01,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:01,625 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:01,627 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:01,627 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,627 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,627 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:01,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:01,629 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:01,629 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:01,629 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:01,629 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:01,629 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:01,630 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,630 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,630 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:01,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:01,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:01,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,632 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:01,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:01,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:01,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:01,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:01,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:01,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:01,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:01,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:01,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,634 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:01,634 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:01,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:01,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858561644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:01,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858561667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:01,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858561667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:01,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858561667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:01,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858561668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,734 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:01,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:01,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:01,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:01,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:01,737 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:01,738 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:01,738 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:01,738 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:01,738 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,738 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,740 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,740 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:01,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:01,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:01,741 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,741 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:01,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:01,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:01,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:01,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:01,742 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:01,742 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:01,742 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:01,742 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:01,742 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:01,742 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:01,742 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:01,742 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:55:01,742 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:01,743 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:01,743 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:01,743 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,743 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:01,743 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:01,748 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,748 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,748 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,748 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:01,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e085e3[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:01,749 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:01,750 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,751 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:01,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:01,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:01,751 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,751 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:01,752 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,752 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:01,753 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:01,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,753 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:01,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:01,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:01,755 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:01,755 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:01,755 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:01,755 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:01,755 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:01,759 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,759 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:01,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,759 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:01,759 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,760 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,762 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,762 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,762 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,762 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:01,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:01,763 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:01,763 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:01,763 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,763 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,763 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,762 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:01,763 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:01,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:01,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:01,763 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:01,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:01,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:01,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:01,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:01,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858561770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:01,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58720 deadline: 1681858561772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:01,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858561772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:01,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858561772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:01,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858561773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,863 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:01,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:01,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:01,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:01,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:01,866 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:01,867 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:01,867 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:01,867 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:01,867 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,867 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,868 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:01,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:01,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:01,868 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,869 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,869 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:01,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:01,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:01,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:01,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:01,872 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:01,872 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:01,872 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:01,872 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:01,872 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:01,872 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:01,872 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:01,872 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:55:01,872 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:01,872 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:01,872 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:01,872 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,872 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:01,872 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:01,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,874 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:01,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ff58e2[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:01,876 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:01,876 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,878 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,878 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:01,878 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:01,878 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:01,878 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,878 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:01,879 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,879 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:01,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,880 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:01,880 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:01,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:01,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:01,882 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:01,882 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:01,882 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:01,882 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:01,882 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:01,884 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,884 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:01,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:01,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,884 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:01,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:01,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:01,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:01,885 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,887 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:01,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:01,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:01,890 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:01,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:01,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:01,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:01,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:01,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:01,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,890 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:01,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:01,890 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:01,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:01,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:01,891 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:01,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858561973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:01,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58720 deadline: 1681858561975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:01,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58742 deadline: 1681858561975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:01,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858561975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:01,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858561976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:01,988 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/48e20d1e888045f6b90c0f3e071ddbff 2023-04-18 22:55:01,991 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:01,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:01,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:01,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:01,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:01,999 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:01,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:01,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:01,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:01,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:01,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,002 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:02,002 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:02,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:02,002 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,002 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:02,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:02,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:02,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:02,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:02,003 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:02,003 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:02,003 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:02,003 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:02,003 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:02,003 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:02,003 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:02,003 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:55:02,003 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:02,004 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:02,004 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:02,004 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,004 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:02,004 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:02,005 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,005 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:02,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:02,006 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,006 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:02,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:02,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dd0f2f7[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:02,007 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:02,007 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,008 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,008 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:02,008 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:02,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:02,008 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,009 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:02,010 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,010 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:02,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:02,012 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:02,012 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:02,012 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:02,012 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:02,012 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:02,015 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,015 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:02,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:02,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,015 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:02,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:02,016 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,017 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,019 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:02,019 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:02,020 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:02,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:02,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:02,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:02,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:02,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,020 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:02,020 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:02,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:02,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:02,120 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:02,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:02,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:02,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:02,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:02,124 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:02,124 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:02,124 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:02,124 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:02,124 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,124 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,125 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,125 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:02,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:02,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:02,126 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,126 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:02,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:02,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:02,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:02,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:02,127 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:02,127 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:02,127 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:02,127 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:02,127 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:02,127 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:02,128 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:02,128 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:55:02,128 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:02,128 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:02,128 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:02,128 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,128 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:02,128 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:02,130 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,130 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:02,130 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,130 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:02,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:02,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:02,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4122e36f[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:02,132 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:02,132 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,137 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,137 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:02,137 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:02,137 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,138 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:02,138 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:02,139 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,139 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:02,139 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,139 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:02,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:02,142 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:02,142 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:02,142 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:02,142 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:02,142 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:02,143 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,143 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:02,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,143 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:02,144 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:02,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:02,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:02,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:02,146 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:02,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,149 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,149 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,149 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,149 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:02,149 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,149 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,149 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,149 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:02,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:02,150 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:02,150 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:02,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:02,150 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:02,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:02,150 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:02,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:02,250 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:02,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:02,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:02,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:02,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:02,253 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:02,253 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:02,254 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:02,254 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:02,254 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,254 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,256 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,256 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:02,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:02,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:02,256 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,256 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:02,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:02,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:02,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:02,257 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:02,257 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:02,257 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:55:02,258 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:02,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:02,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:02,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:02,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:02,259 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,259 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:02,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:02,259 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,259 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:02,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:02,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:02,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:02,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:02,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40eca935[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:02,261 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:02,261 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,263 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,263 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,263 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,263 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:02,263 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:02,263 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:02,266 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:02,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:02,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:02,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:02,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:02,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:02,268 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:02,268 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:02,268 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:02,269 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:02,269 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:02,270 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,271 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:02,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,271 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:02,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,272 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,273 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:02,276 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:02,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:02,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:02,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:02,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:02,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:02,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,276 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:02,277 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:02,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:02,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:02,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:02,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58768 deadline: 1681858562277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:02,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1681858562279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:02,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858562280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:02,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58720 deadline: 1681858562280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:02,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858562283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,377 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:02,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:02,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:02,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:02,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:02,380 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:02,380 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:02,380 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:02,380 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:02,381 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,381 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,382 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:02,382 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:02,383 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,383 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:02,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:02,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:02,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:02,384 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:02,384 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:02,384 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:02,384 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:02,384 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:02,384 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:02,384 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:02,384 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:55:02,384 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:02,384 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:02,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:02,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:02,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:02,387 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,387 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:02,387 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:02,387 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:02,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:02,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:02,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:02,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f6bc182[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:02,389 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:02,389 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,391 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,391 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:02,391 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:02,391 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:02,391 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,391 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:02,398 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:02,398 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,398 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:02,398 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:02,399 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/7776cf08abcb4936a999c1939d7c5c4c 2023-04-18 22:55:02,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:02,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:02,402 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:02,402 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:02,402 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:02,402 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:02,402 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:02,403 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/64ea5428b48a4bd98c6b98330b61e344 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/64ea5428b48a4bd98c6b98330b61e344 2023-04-18 22:55:02,404 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:02,404 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,404 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:02,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,404 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:02,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,407 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:02,408 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/64ea5428b48a4bd98c6b98330b61e344, entries=150, sequenceid=459, filesize=11.9 K 2023-04-18 22:55:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/48e20d1e888045f6b90c0f3e071ddbff as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/48e20d1e888045f6b90c0f3e071ddbff 2023-04-18 22:55:02,410 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,410 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,410 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:02,410 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,410 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,411 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,410 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:02,411 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:02,411 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:02,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:02,411 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:02,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:02,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:02,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:02,411 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:02,411 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:02,412 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/48e20d1e888045f6b90c0f3e071ddbff, entries=150, sequenceid=459, filesize=11.9 K 2023-04-18 22:55:02,413 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/7776cf08abcb4936a999c1939d7c5c4c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/7776cf08abcb4936a999c1939d7c5c4c 2023-04-18 22:55:02,416 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/7776cf08abcb4936a999c1939d7c5c4c, entries=150, sequenceid=459, filesize=11.9 K 2023-04-18 22:55:02,417 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 94dcab632e7fc6a2d56c5605ce1488fa in 862ms, sequenceid=459, compaction requested=true 2023-04-18 22:55:02,417 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:02,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:02,417 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:02,417 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:02,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:02,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:02,418 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:02,418 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:02,418 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/A is initiating minor compaction (all files) 2023-04-18 22:55:02,418 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/B is initiating minor compaction (all files) 2023-04-18 22:55:02,418 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/A in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:02,418 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/B in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:02,418 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8db234c322b24c6db8c6d5b5ff0b6e15, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d1abdc83f2cf4b1b8c14ddd0fb7bcf83, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/64ea5428b48a4bd98c6b98330b61e344] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=36.5 K 2023-04-18 22:55:02,418 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/882fb29e927045a2835fa72273f5740a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/1f8d71a9b8004c16877e7eed9a28f00d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/48e20d1e888045f6b90c0f3e071ddbff] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=36.5 K 2023-04-18 22:55:02,418 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 8db234c322b24c6db8c6d5b5ff0b6e15, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1681858500295 2023-04-18 22:55:02,418 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 882fb29e927045a2835fa72273f5740a, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1681858500295 2023-04-18 22:55:02,419 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting d1abdc83f2cf4b1b8c14ddd0fb7bcf83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681858500419 2023-04-18 22:55:02,419 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 1f8d71a9b8004c16877e7eed9a28f00d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681858500419 2023-04-18 22:55:02,419 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 64ea5428b48a4bd98c6b98330b61e344, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681858501550 2023-04-18 22:55:02,419 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 48e20d1e888045f6b90c0f3e071ddbff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681858501550 2023-04-18 22:55:02,425 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#A#compaction#296 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:02,428 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#B#compaction#297 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:02,456 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/52598d9e51e14f3a9390bdf24c80f2dc as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/52598d9e51e14f3a9390bdf24c80f2dc 2023-04-18 22:55:02,460 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/B of 94dcab632e7fc6a2d56c5605ce1488fa into 52598d9e51e14f3a9390bdf24c80f2dc(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:02,460 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:02,460 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/B, priority=13, startTime=1681858502417; duration=0sec 2023-04-18 22:55:02,460 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:02,460 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:02,462 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:02,462 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/C is initiating minor compaction (all files) 2023-04-18 22:55:02,462 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/C in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:02,462 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/3ce970096b264f9f95c586b558c51af8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/5ead824a7dc54232bb772d0361a74657, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/7776cf08abcb4936a999c1939d7c5c4c] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=36.5 K 2023-04-18 22:55:02,463 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 3ce970096b264f9f95c586b558c51af8, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1681858500295 2023-04-18 22:55:02,463 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 5ead824a7dc54232bb772d0361a74657, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681858500419 2023-04-18 22:55:02,463 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 7776cf08abcb4936a999c1939d7c5c4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681858501550 2023-04-18 22:55:02,469 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#C#compaction#298 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:02,511 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:02,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:02,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:02,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:02,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:02,517 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:02,517 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:02,517 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:02,517 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:02,517 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,517 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,519 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:02,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:02,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:02,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:02,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:02,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:02,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:02,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:02,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:02,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:02,520 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:02,520 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:02,521 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:02,521 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:02,521 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:02,521 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:02,521 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:02,522 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-18 22:55:02,522 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:55:02,522 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:02,522 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:55:02,522 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:02,522 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:55:02,522 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:02,530 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/8ff9a114a55740a5b9d70e8b9fbf8e2e 2023-04-18 22:55:02,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:55:02,783 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:55:02,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:02,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:02,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58720 deadline: 1681858562788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858562788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:02,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58768 deadline: 1681858562788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:02,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858562789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:02,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1681858562791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,843 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/73198fbd145c4b99bafd506da2491e30 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/73198fbd145c4b99bafd506da2491e30 2023-04-18 22:55:02,848 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/A of 94dcab632e7fc6a2d56c5605ce1488fa into 73198fbd145c4b99bafd506da2491e30(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:02,849 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:02,849 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/A, priority=13, startTime=1681858502417; duration=0sec 2023-04-18 22:55:02,849 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:02,878 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/6c2e57de49da4342be094d2354acf952 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/6c2e57de49da4342be094d2354acf952 2023-04-18 22:55:02,884 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/C of 94dcab632e7fc6a2d56c5605ce1488fa into 6c2e57de49da4342be094d2354acf952(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:02,884 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:02,884 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/C, priority=13, startTime=1681858502417; duration=0sec 2023-04-18 22:55:02,884 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:02,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:02,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858562892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:02,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58768 deadline: 1681858562892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:02,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858562892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:02,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1681858562896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:02,944 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/8323b2b4582b430aaf2e843d39d9053a 2023-04-18 22:55:03,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858563094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858563094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58768 deadline: 1681858563095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1681858563099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,362 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/f505308b61ef4a44a5c3b75a8449c857 2023-04-18 22:55:03,366 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/8ff9a114a55740a5b9d70e8b9fbf8e2e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8ff9a114a55740a5b9d70e8b9fbf8e2e 2023-04-18 22:55:03,369 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8ff9a114a55740a5b9d70e8b9fbf8e2e, entries=150, sequenceid=497, filesize=11.9 K 2023-04-18 22:55:03,370 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/8323b2b4582b430aaf2e843d39d9053a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/8323b2b4582b430aaf2e843d39d9053a 2023-04-18 22:55:03,373 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/8323b2b4582b430aaf2e843d39d9053a, entries=150, sequenceid=497, filesize=11.9 K 2023-04-18 22:55:03,374 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/f505308b61ef4a44a5c3b75a8449c857 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/f505308b61ef4a44a5c3b75a8449c857 2023-04-18 22:55:03,385 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/f505308b61ef4a44a5c3b75a8449c857, entries=150, sequenceid=497, filesize=11.9 K 2023-04-18 22:55:03,385 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 94dcab632e7fc6a2d56c5605ce1488fa in 863ms, sequenceid=497, compaction requested=false 2023-04-18 22:55:03,386 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:03,386 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,391 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:03,391 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:03,392 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,392 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:03,393 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:03,393 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:03,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:03,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7526c3d2[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:03,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:03,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:03,396 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:03,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:03,396 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:03,396 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:03,396 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:03,397 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,397 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:03,399 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:03,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:03,399 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:03,399 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:03,399 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:03,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:03,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:55:03,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:03,400 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:55:03,400 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:55:03,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:03,400 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:03,400 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:55:03,400 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:03,400 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:55:03,400 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:03,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:03,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:03,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:03,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:03,402 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:03,402 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:03,402 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:03,402 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:03,402 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:03,407 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:03,407 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:03,407 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:03,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:03,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:03,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:03,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:03,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:03,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:03,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:03,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:03,408 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,410 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,411 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/516cf2be7bc04fcfa0015c92e663cb2f 2023-04-18 22:55:03,413 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,413 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:03,413 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:03,413 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:03,413 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:03,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:03,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:03,413 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,413 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:03,413 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:03,413 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:03,413 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:03,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:03,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:03,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:03,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:03,414 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:03,414 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:03,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858503417, max timestamp=1681858503422Num uniques 859; 2023-04-18 22:55:03,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:55:03,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858503417, max timestamp=1681858503422Num uniques 859; 2023-04-18 22:55:03,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:55:03,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858503417, max timestamp=1681858503422Num uniques 859; 2023-04-18 22:55:03,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:55:03,432 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/10ec4eceb5c7437ca78cb482a98a0630 2023-04-18 22:55:03,442 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/c1eea17a9bbf441f98ce2fc14ee2d91c 2023-04-18 22:55:03,447 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/516cf2be7bc04fcfa0015c92e663cb2f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/516cf2be7bc04fcfa0015c92e663cb2f 2023-04-18 22:55:03,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1681858563446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58768 deadline: 1681858563446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,452 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/516cf2be7bc04fcfa0015c92e663cb2f, entries=150, sequenceid=509, filesize=11.9 K 2023-04-18 22:55:03,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858563449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58762 deadline: 1681858563449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,453 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/10ec4eceb5c7437ca78cb482a98a0630 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/10ec4eceb5c7437ca78cb482a98a0630 2023-04-18 22:55:03,457 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/10ec4eceb5c7437ca78cb482a98a0630, entries=150, sequenceid=509, filesize=11.9 K 2023-04-18 22:55:03,458 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/c1eea17a9bbf441f98ce2fc14ee2d91c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/c1eea17a9bbf441f98ce2fc14ee2d91c 2023-04-18 22:55:03,462 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/c1eea17a9bbf441f98ce2fc14ee2d91c, entries=150, sequenceid=509, filesize=11.9 K 2023-04-18 22:55:03,462 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 94dcab632e7fc6a2d56c5605ce1488fa in 63ms, sequenceid=509, compaction requested=true 2023-04-18 22:55:03,463 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:03,463 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:03,463 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:03,463 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:03,463 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:03,463 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:03,464 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:03,464 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/A is initiating minor compaction (all files) 2023-04-18 22:55:03,464 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:03,464 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/A in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:03,464 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/B is initiating minor compaction (all files) 2023-04-18 22:55:03,464 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/73198fbd145c4b99bafd506da2491e30, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8ff9a114a55740a5b9d70e8b9fbf8e2e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/516cf2be7bc04fcfa0015c92e663cb2f] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=36.6 K 2023-04-18 22:55:03,464 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/B in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:03,464 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/52598d9e51e14f3a9390bdf24c80f2dc, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/8323b2b4582b430aaf2e843d39d9053a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/10ec4eceb5c7437ca78cb482a98a0630] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=36.6 K 2023-04-18 22:55:03,465 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 73198fbd145c4b99bafd506da2491e30, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681858501550 2023-04-18 22:55:03,465 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 52598d9e51e14f3a9390bdf24c80f2dc, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681858501550 2023-04-18 22:55:03,465 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 8ff9a114a55740a5b9d70e8b9fbf8e2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1681858501644 2023-04-18 22:55:03,465 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 8323b2b4582b430aaf2e843d39d9053a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1681858501644 2023-04-18 22:55:03,465 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 516cf2be7bc04fcfa0015c92e663cb2f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681858502787 2023-04-18 22:55:03,466 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 10ec4eceb5c7437ca78cb482a98a0630, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681858502787 2023-04-18 22:55:03,472 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#A#compaction#305 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:03,473 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#B#compaction#306 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:03,514 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:03,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:03,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:03,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:03,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:03,517 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:03,517 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:03,517 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:03,517 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:03,518 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:03,518 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:03,519 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:03,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:03,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:03,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:03,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:03,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:03,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:03,521 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:03,521 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:03,521 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:03,521 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:03,521 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:03,521 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:03,521 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:03,521 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-18 22:55:03,521 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:55:03,521 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:03,521 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:55:03,522 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:03,522 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:55:03,522 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:03,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:55:03,553 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. as already flushing 2023-04-18 22:55:03,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58768 deadline: 1681858563559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1681858563559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858563562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58762 deadline: 1681858563563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58768 deadline: 1681858563664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1681858563664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858563668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58762 deadline: 1681858563668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58720 deadline: 1681858563796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58768 deadline: 1681858563867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1681858563868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858563870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:03,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58762 deadline: 1681858563871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:03,890 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/26df935fe64d4b06a061d426495750d6 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/26df935fe64d4b06a061d426495750d6 2023-04-18 22:55:03,894 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/8cb230e4b348482b845c2521920d0473 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8cb230e4b348482b845c2521920d0473 2023-04-18 22:55:03,898 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/A of 94dcab632e7fc6a2d56c5605ce1488fa into 8cb230e4b348482b845c2521920d0473(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:03,898 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:03,898 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/A, priority=13, startTime=1681858503463; duration=0sec 2023-04-18 22:55:03,898 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:03,898 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:03,899 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:03,899 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 94dcab632e7fc6a2d56c5605ce1488fa/C is initiating minor compaction (all files) 2023-04-18 22:55:03,899 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94dcab632e7fc6a2d56c5605ce1488fa/C in TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:03,900 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/6c2e57de49da4342be094d2354acf952, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/f505308b61ef4a44a5c3b75a8449c857, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/c1eea17a9bbf441f98ce2fc14ee2d91c] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp, totalSize=36.6 K 2023-04-18 22:55:03,900 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/B of 94dcab632e7fc6a2d56c5605ce1488fa into 26df935fe64d4b06a061d426495750d6(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:03,900 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 6c2e57de49da4342be094d2354acf952, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681858501550 2023-04-18 22:55:03,900 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:03,900 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/B, priority=13, startTime=1681858503463; duration=0sec 2023-04-18 22:55:03,900 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:03,900 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting f505308b61ef4a44a5c3b75a8449c857, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1681858501644 2023-04-18 22:55:03,901 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting c1eea17a9bbf441f98ce2fc14ee2d91c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1681858502787 2023-04-18 22:55:03,908 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94dcab632e7fc6a2d56c5605ce1488fa#C#compaction#308 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:03,929 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/8273138e4ecc43c59ea6addf93986728 2023-04-18 22:55:04,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:04,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58768 deadline: 1681858564169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:04,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1681858564172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:04,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858564174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:04,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58762 deadline: 1681858564176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,317 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/b4977edd32cb4c15bc33059e6baafe15 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/b4977edd32cb4c15bc33059e6baafe15 2023-04-18 22:55:04,321 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94dcab632e7fc6a2d56c5605ce1488fa/C of 94dcab632e7fc6a2d56c5605ce1488fa into b4977edd32cb4c15bc33059e6baafe15(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:04,321 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:04,321 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa., storeName=94dcab632e7fc6a2d56c5605ce1488fa/C, priority=13, startTime=1681858503463; duration=0sec 2023-04-18 22:55:04,321 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:04,339 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/35a3fd1685b54fa4a6e222d5aec24c9b 2023-04-18 22:55:04,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:04,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58768 deadline: 1681858564672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:04,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58742 deadline: 1681858564677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:04,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58762 deadline: 1681858564678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:04,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58758 deadline: 1681858564680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,748 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/efa5dff680274e8dbb69027375fbb3b8 2023-04-18 22:55:04,754 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/8273138e4ecc43c59ea6addf93986728 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8273138e4ecc43c59ea6addf93986728 2023-04-18 22:55:04,757 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8273138e4ecc43c59ea6addf93986728, entries=150, sequenceid=546, filesize=11.9 K 2023-04-18 22:55:04,765 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/35a3fd1685b54fa4a6e222d5aec24c9b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/35a3fd1685b54fa4a6e222d5aec24c9b 2023-04-18 22:55:04,768 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/35a3fd1685b54fa4a6e222d5aec24c9b, entries=150, sequenceid=546, filesize=11.9 K 2023-04-18 22:55:04,769 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/efa5dff680274e8dbb69027375fbb3b8 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/efa5dff680274e8dbb69027375fbb3b8 2023-04-18 22:55:04,773 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/efa5dff680274e8dbb69027375fbb3b8, entries=150, sequenceid=546, filesize=11.9 K 2023-04-18 22:55:04,773 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for 94dcab632e7fc6a2d56c5605ce1488fa in 1252ms, sequenceid=546, compaction requested=false 2023-04-18 22:55:04,774 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:04,774 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:04,774 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:04,774 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:04,774 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,774 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:04,774 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:04,775 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:04,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:04,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:04,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:04,776 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:04,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:04,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:04,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:04,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:04,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23c96f82[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:04,778 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:04,778 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:04,780 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:04,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:04,780 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:04,780 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:04,780 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:04,781 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,781 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:04,782 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:04,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:04,784 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:04,784 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:04,785 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:04,785 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:04,785 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:04,786 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:04,786 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:04,786 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:04,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:04,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:04,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:04,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:04,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:04,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:04,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:04,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:04,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:04,787 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:04,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:04,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:04,788 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:04,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:04,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,790 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,790 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:04,790 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:04,790 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,790 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:04,790 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:04,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:04,791 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:04,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:04,791 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:04,791 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:04,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:04,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:04,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:04,791 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:04,791 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:04,791 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:04,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:04,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:04,891 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:04,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:04,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:04,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:04,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:04,895 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:04,895 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:04,895 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:04,895 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:04,895 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:04,895 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:04,898 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,898 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:04,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:04,898 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:04,898 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:04,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:04,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:04,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:04,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:04,899 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:04,899 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:04,900 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:04,900 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:04,900 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:04,900 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:04,900 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. started... 2023-04-18 22:55:04,900 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-18 22:55:04,900 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:55:04,900 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:04,900 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:55:04,900 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:04,900 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:55:04,900 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:04,909 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/901d440b7e3642d194c9c15d77cf2f73 2023-04-18 22:55:04,921 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/cda72bf22b6f49f3bac3b964bbb6c4ec 2023-04-18 22:55:05,069 DEBUG [Thread-1769] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x434e0eab to 127.0.0.1:50825 2023-04-18 22:55:05,069 DEBUG [Thread-1769] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:05,069 DEBUG [Thread-1767] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dea0f9b to 127.0.0.1:50825 2023-04-18 22:55:05,069 DEBUG [Thread-1767] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:05,070 DEBUG [Thread-1765] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x753156b2 to 127.0.0.1:50825 2023-04-18 22:55:05,070 DEBUG [Thread-1765] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:05,070 DEBUG [Thread-1773] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de064cb to 127.0.0.1:50825 2023-04-18 22:55:05,070 DEBUG [Thread-1771] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c43dddd to 127.0.0.1:50825 2023-04-18 22:55:05,070 DEBUG [Thread-1773] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:05,070 DEBUG [Thread-1771] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:05,332 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/222bf10e7a434564ac9803506c427546 2023-04-18 22:55:05,335 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/901d440b7e3642d194c9c15d77cf2f73 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/901d440b7e3642d194c9c15d77cf2f73 2023-04-18 22:55:05,338 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/901d440b7e3642d194c9c15d77cf2f73, entries=150, sequenceid=560, filesize=11.9 K 2023-04-18 22:55:05,339 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/cda72bf22b6f49f3bac3b964bbb6c4ec as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/cda72bf22b6f49f3bac3b964bbb6c4ec 2023-04-18 22:55:05,342 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/cda72bf22b6f49f3bac3b964bbb6c4ec, entries=150, sequenceid=560, filesize=11.9 K 2023-04-18 22:55:05,343 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/222bf10e7a434564ac9803506c427546 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/222bf10e7a434564ac9803506c427546 2023-04-18 22:55:05,346 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/222bf10e7a434564ac9803506c427546, entries=150, sequenceid=560, filesize=11.9 K 2023-04-18 22:55:05,347 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 94dcab632e7fc6a2d56c5605ce1488fa in 447ms, sequenceid=560, compaction requested=true 2023-04-18 22:55:05,347 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:05,347 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:05,347 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:05,347 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:05,347 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:05,347 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:05,347 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:05,349 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:05,349 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:05,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:05,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:05,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:05,349 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:05,349 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:05,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:05,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:05,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:05,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:05,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:05,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:05,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c9dfcf[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:05,350 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:05,350 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:05,352 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:05,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:05,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:05,352 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:05,352 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:05,352 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:05,352 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:05,352 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:05,354 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:05,354 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:05,354 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:05,354 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:05,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:05,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:05,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:05,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:05,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:05,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:05,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:05,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:05,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:05,356 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:05,356 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:05,356 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:05,356 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:05,356 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:05,358 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:05,358 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:05,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:05,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:05,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:05,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:05,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:05,358 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:05,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:05,358 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:05,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:05,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:05,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:05,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:05,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:05,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:05,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:05,359 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:05,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:05,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:05,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:05,361 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:05,361 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:05,361 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:05,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:05,361 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:05,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:05,362 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:05,361 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:05,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:05,362 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:05,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:05,362 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:05,362 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:05,362 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:05,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:05,362 DEBUG [Thread-1764] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:05,362 DEBUG [Thread-1764] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:05,462 DEBUG [Thread-1764] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:05,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:05,677 DEBUG [Thread-1758] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f4ad99d to 127.0.0.1:50825 2023-04-18 22:55:05,677 DEBUG [Thread-1758] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:05,686 DEBUG [Thread-1754] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70043cf0 to 127.0.0.1:50825 2023-04-18 22:55:05,686 DEBUG [Thread-1756] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b7fed0 to 127.0.0.1:50825 2023-04-18 22:55:05,686 DEBUG [Thread-1754] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:05,686 DEBUG [Thread-1756] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:05,691 DEBUG [Thread-1762] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7810a231 to 127.0.0.1:50825 2023-04-18 22:55:05,691 DEBUG [Thread-1762] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:05,818 DEBUG [Thread-1760] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x330b77da to 127.0.0.1:50825 2023-04-18 22:55:05,818 DEBUG [Thread-1760] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:05,818 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-18 22:55:05,819 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 58 2023-04-18 22:55:05,819 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 52 2023-04-18 22:55:05,819 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 58 2023-04-18 22:55:05,819 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 49 2023-04-18 22:55:05,819 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 250 2023-04-18 22:55:05,819 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-18 22:55:05,819 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(398): read 5782 2023-04-18 22:55:05,819 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(398): read 6012 2023-04-18 22:55:05,819 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(398): read 6099 2023-04-18 22:55:05,819 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(398): read 5808 2023-04-18 22:55:05,819 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(398): read 6030 2023-04-18 22:55:05,819 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-18 22:55:05,819 INFO [Listener at localhost/36651] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-18 22:55:05,819 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ed1dde9 to 127.0.0.1:50825 2023-04-18 22:55:05,819 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:05,820 INFO [Listener at localhost/36651] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-18 22:55:05,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-18 22:55:05,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:05,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-18 22:55:05,823 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858505823"}]},"ts":"1681858505823"} 2023-04-18 22:55:05,824 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-18 22:55:05,826 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-18 22:55:05,826 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=94dcab632e7fc6a2d56c5605ce1488fa, UNASSIGN}] 2023-04-18 22:55:05,827 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=94dcab632e7fc6a2d56c5605ce1488fa, UNASSIGN 2023-04-18 22:55:05,828 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=94dcab632e7fc6a2d56c5605ce1488fa, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:05,828 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858505828"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858505828"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858505828"}]},"ts":"1681858505828"} 2023-04-18 22:55:05,829 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:55:05,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-18 22:55:05,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:55:05,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 94dcab632e7fc6a2d56c5605ce1488fa, disabling compactions & flushes 2023-04-18 22:55:05,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:05,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:05,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. after waiting 0 ms 2023-04-18 22:55:05,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:05,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 94dcab632e7fc6a2d56c5605ce1488fa 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-18 22:55:05,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=A 2023-04-18 22:55:05,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:05,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=B 2023-04-18 22:55:05,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:05,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94dcab632e7fc6a2d56c5605ce1488fa, store=C 2023-04-18 22:55:05,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:06,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-18 22:55:06,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/ea997b5deab548e7877f4235b40be7fa 2023-04-18 22:55:06,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-18 22:55:06,798 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/61d53d34cf3e4cb1b8d05fd324e3030c 2023-04-18 22:55:06,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-18 22:55:07,207 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/1e60045075c0437888f452d0b7c4697f 2023-04-18 22:55:07,211 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/A/ea997b5deab548e7877f4235b40be7fa as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/ea997b5deab548e7877f4235b40be7fa 2023-04-18 22:55:07,214 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/ea997b5deab548e7877f4235b40be7fa, entries=100, sequenceid=568, filesize=9.5 K 2023-04-18 22:55:07,214 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/B/61d53d34cf3e4cb1b8d05fd324e3030c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/61d53d34cf3e4cb1b8d05fd324e3030c 2023-04-18 22:55:07,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/61d53d34cf3e4cb1b8d05fd324e3030c, entries=100, sequenceid=568, filesize=9.5 K 2023-04-18 22:55:07,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/.tmp/C/1e60045075c0437888f452d0b7c4697f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/1e60045075c0437888f452d0b7c4697f 2023-04-18 22:55:07,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/1e60045075c0437888f452d0b7c4697f, entries=100, sequenceid=568, filesize=9.5 K 2023-04-18 22:55:07,222 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 94dcab632e7fc6a2d56c5605ce1488fa in 1240ms, sequenceid=568, compaction requested=true 2023-04-18 22:55:07,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/ce117ff783c04a5ca2cb8ecdc4d34379, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/14a1c6d11aa44a6c89852774a85a2d1c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/f92da3cc068240d98a65031c0e765498, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/34e0d5d49321438faf642d9050df67b1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/03896dbd08724f8abb6ccfb22b3fd219, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/4c54963eaf5143eeae7fc07cb93f3312, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/78b7675f5dc04a4caf2873fb3fb169ea, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/b20df012e0264a84b3dada148d076698, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/0db958221c964a768bb91e69ec6b83aa, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8bd31e02f33b4fe4b83e9b26f17deef5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/03a66cfc2af245d58989e21fc78db7be, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8801703f3f944450ab672f2f8b11f498, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/1e604fc9d4a149cea0690af542a248db, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/690477a63cc647d695d2898ff398bbe3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/e888b495ecf64b279bbe64d4a00fc9cb, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/918c6daa773f4f039352864a71874aa4, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d8fef93ede284e9ebff73606fed07a29, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/a8d43003feac47dbb45f0d9cdaedd7c4, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/a9e91e8c25e749fdb90b0058f2542925, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/c79c933be7bc4b69a623cdbc467fcacf, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/88a0763066ef4bcf84099cbe32e64a51, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d4d6b3870a0a414491000c7cac70577f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/ca6151c3d4714a988b1ed18da5782a2d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8db234c322b24c6db8c6d5b5ff0b6e15, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d1abdc83f2cf4b1b8c14ddd0fb7bcf83, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/73198fbd145c4b99bafd506da2491e30, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/64ea5428b48a4bd98c6b98330b61e344, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8ff9a114a55740a5b9d70e8b9fbf8e2e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/516cf2be7bc04fcfa0015c92e663cb2f] to archive 2023-04-18 22:55:07,245 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:55:07,246 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/ce117ff783c04a5ca2cb8ecdc4d34379 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/ce117ff783c04a5ca2cb8ecdc4d34379 2023-04-18 22:55:07,247 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/14a1c6d11aa44a6c89852774a85a2d1c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/14a1c6d11aa44a6c89852774a85a2d1c 2023-04-18 22:55:07,248 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/f92da3cc068240d98a65031c0e765498 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/f92da3cc068240d98a65031c0e765498 2023-04-18 22:55:07,249 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/34e0d5d49321438faf642d9050df67b1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/34e0d5d49321438faf642d9050df67b1 2023-04-18 22:55:07,250 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/03896dbd08724f8abb6ccfb22b3fd219 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/03896dbd08724f8abb6ccfb22b3fd219 2023-04-18 22:55:07,251 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/4c54963eaf5143eeae7fc07cb93f3312 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/4c54963eaf5143eeae7fc07cb93f3312 2023-04-18 22:55:07,252 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/78b7675f5dc04a4caf2873fb3fb169ea to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/78b7675f5dc04a4caf2873fb3fb169ea 2023-04-18 22:55:07,253 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/b20df012e0264a84b3dada148d076698 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/b20df012e0264a84b3dada148d076698 2023-04-18 22:55:07,253 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/0db958221c964a768bb91e69ec6b83aa to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/0db958221c964a768bb91e69ec6b83aa 2023-04-18 22:55:07,254 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8bd31e02f33b4fe4b83e9b26f17deef5 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8bd31e02f33b4fe4b83e9b26f17deef5 2023-04-18 22:55:07,255 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/03a66cfc2af245d58989e21fc78db7be to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/03a66cfc2af245d58989e21fc78db7be 2023-04-18 22:55:07,256 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8801703f3f944450ab672f2f8b11f498 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8801703f3f944450ab672f2f8b11f498 2023-04-18 22:55:07,257 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/1e604fc9d4a149cea0690af542a248db to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/1e604fc9d4a149cea0690af542a248db 2023-04-18 22:55:07,258 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/690477a63cc647d695d2898ff398bbe3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/690477a63cc647d695d2898ff398bbe3 2023-04-18 22:55:07,259 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/e888b495ecf64b279bbe64d4a00fc9cb to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/e888b495ecf64b279bbe64d4a00fc9cb 2023-04-18 22:55:07,260 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/918c6daa773f4f039352864a71874aa4 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/918c6daa773f4f039352864a71874aa4 2023-04-18 22:55:07,260 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d8fef93ede284e9ebff73606fed07a29 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d8fef93ede284e9ebff73606fed07a29 2023-04-18 22:55:07,261 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/a8d43003feac47dbb45f0d9cdaedd7c4 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/a8d43003feac47dbb45f0d9cdaedd7c4 2023-04-18 22:55:07,262 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/a9e91e8c25e749fdb90b0058f2542925 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/a9e91e8c25e749fdb90b0058f2542925 2023-04-18 22:55:07,263 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/c79c933be7bc4b69a623cdbc467fcacf to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/c79c933be7bc4b69a623cdbc467fcacf 2023-04-18 22:55:07,264 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/88a0763066ef4bcf84099cbe32e64a51 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/88a0763066ef4bcf84099cbe32e64a51 2023-04-18 22:55:07,265 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d4d6b3870a0a414491000c7cac70577f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d4d6b3870a0a414491000c7cac70577f 2023-04-18 22:55:07,265 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/ca6151c3d4714a988b1ed18da5782a2d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/ca6151c3d4714a988b1ed18da5782a2d 2023-04-18 22:55:07,266 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8db234c322b24c6db8c6d5b5ff0b6e15 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8db234c322b24c6db8c6d5b5ff0b6e15 2023-04-18 22:55:07,267 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d1abdc83f2cf4b1b8c14ddd0fb7bcf83 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/d1abdc83f2cf4b1b8c14ddd0fb7bcf83 2023-04-18 22:55:07,268 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/73198fbd145c4b99bafd506da2491e30 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/73198fbd145c4b99bafd506da2491e30 2023-04-18 22:55:07,269 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/64ea5428b48a4bd98c6b98330b61e344 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/64ea5428b48a4bd98c6b98330b61e344 2023-04-18 22:55:07,270 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8ff9a114a55740a5b9d70e8b9fbf8e2e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8ff9a114a55740a5b9d70e8b9fbf8e2e 2023-04-18 22:55:07,270 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/516cf2be7bc04fcfa0015c92e663cb2f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/516cf2be7bc04fcfa0015c92e663cb2f 2023-04-18 22:55:07,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/5d35722e93944a428fc5ec81efd84cf9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/015e9f9241834650bdfd7246c9395519, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/3c0b761459354fcda65b2956a16c2d21, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/7141cf04913948e8879f05db024e0c6c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/bdd5f7d4c430428f9908e0dbb03fd167, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/1640da04082e432fa2aab20f2a4fb77c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/0d1b4120d3ef44d5a6f06140c11afbbd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/4a5605fac9b0442b98c49c795b322913, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/4d9a5afcc90a49659b93293d090927d9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/a7b5d8b5fc1a4522936f5abda4100b41, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/bf341e6c318941d68b309c917f5f9511, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/7d2ef07ecad24270bacdd3c3c1b9c8f8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/50dc910baf3943bb8bd4457a48215f64, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/d22e0e0335d846d6ad3c0fb57044b818, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/b7c024cc3d3d4bb0a7a0af6a6d8c66dd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/83d5b7b2318047618188f488e5ba40df, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/94e5f331477f4f99ba12b50bc1712b41, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/b9b7afea882e4b7080601a6223728058, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/733397e581204e9dad6baea0fa2ab58b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/580eed59975c4882a675bbacadeebd2e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/a9991a3c4e544f28999e80a3d86172bd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/d1a31c7da85f4d819346e70dd08530e0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/882fb29e927045a2835fa72273f5740a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/fff35dabfefb4ffbad9d3549cd3906a2, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/1f8d71a9b8004c16877e7eed9a28f00d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/52598d9e51e14f3a9390bdf24c80f2dc, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/48e20d1e888045f6b90c0f3e071ddbff, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/8323b2b4582b430aaf2e843d39d9053a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/10ec4eceb5c7437ca78cb482a98a0630] to archive 2023-04-18 22:55:07,274 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:55:07,276 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/5d35722e93944a428fc5ec81efd84cf9 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/5d35722e93944a428fc5ec81efd84cf9 2023-04-18 22:55:07,277 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/015e9f9241834650bdfd7246c9395519 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/015e9f9241834650bdfd7246c9395519 2023-04-18 22:55:07,277 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/3c0b761459354fcda65b2956a16c2d21 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/3c0b761459354fcda65b2956a16c2d21 2023-04-18 22:55:07,278 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/7141cf04913948e8879f05db024e0c6c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/7141cf04913948e8879f05db024e0c6c 2023-04-18 22:55:07,279 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/bdd5f7d4c430428f9908e0dbb03fd167 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/bdd5f7d4c430428f9908e0dbb03fd167 2023-04-18 22:55:07,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/1640da04082e432fa2aab20f2a4fb77c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/1640da04082e432fa2aab20f2a4fb77c 2023-04-18 22:55:07,281 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/0d1b4120d3ef44d5a6f06140c11afbbd to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/0d1b4120d3ef44d5a6f06140c11afbbd 2023-04-18 22:55:07,282 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/4a5605fac9b0442b98c49c795b322913 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/4a5605fac9b0442b98c49c795b322913 2023-04-18 22:55:07,283 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/4d9a5afcc90a49659b93293d090927d9 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/4d9a5afcc90a49659b93293d090927d9 2023-04-18 22:55:07,284 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/a7b5d8b5fc1a4522936f5abda4100b41 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/a7b5d8b5fc1a4522936f5abda4100b41 2023-04-18 22:55:07,285 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/bf341e6c318941d68b309c917f5f9511 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/bf341e6c318941d68b309c917f5f9511 2023-04-18 22:55:07,286 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/7d2ef07ecad24270bacdd3c3c1b9c8f8 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/7d2ef07ecad24270bacdd3c3c1b9c8f8 2023-04-18 22:55:07,288 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/50dc910baf3943bb8bd4457a48215f64 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/50dc910baf3943bb8bd4457a48215f64 2023-04-18 22:55:07,289 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/d22e0e0335d846d6ad3c0fb57044b818 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/d22e0e0335d846d6ad3c0fb57044b818 2023-04-18 22:55:07,290 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/b7c024cc3d3d4bb0a7a0af6a6d8c66dd to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/b7c024cc3d3d4bb0a7a0af6a6d8c66dd 2023-04-18 22:55:07,292 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/83d5b7b2318047618188f488e5ba40df to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/83d5b7b2318047618188f488e5ba40df 2023-04-18 22:55:07,293 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/94e5f331477f4f99ba12b50bc1712b41 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/94e5f331477f4f99ba12b50bc1712b41 2023-04-18 22:55:07,294 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/b9b7afea882e4b7080601a6223728058 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/b9b7afea882e4b7080601a6223728058 2023-04-18 22:55:07,296 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/733397e581204e9dad6baea0fa2ab58b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/733397e581204e9dad6baea0fa2ab58b 2023-04-18 22:55:07,297 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/580eed59975c4882a675bbacadeebd2e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/580eed59975c4882a675bbacadeebd2e 2023-04-18 22:55:07,298 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/a9991a3c4e544f28999e80a3d86172bd to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/a9991a3c4e544f28999e80a3d86172bd 2023-04-18 22:55:07,299 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/d1a31c7da85f4d819346e70dd08530e0 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/d1a31c7da85f4d819346e70dd08530e0 2023-04-18 22:55:07,300 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/882fb29e927045a2835fa72273f5740a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/882fb29e927045a2835fa72273f5740a 2023-04-18 22:55:07,301 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/fff35dabfefb4ffbad9d3549cd3906a2 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/fff35dabfefb4ffbad9d3549cd3906a2 2023-04-18 22:55:07,302 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/1f8d71a9b8004c16877e7eed9a28f00d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/1f8d71a9b8004c16877e7eed9a28f00d 2023-04-18 22:55:07,303 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/52598d9e51e14f3a9390bdf24c80f2dc to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/52598d9e51e14f3a9390bdf24c80f2dc 2023-04-18 22:55:07,304 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/48e20d1e888045f6b90c0f3e071ddbff to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/48e20d1e888045f6b90c0f3e071ddbff 2023-04-18 22:55:07,305 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/8323b2b4582b430aaf2e843d39d9053a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/8323b2b4582b430aaf2e843d39d9053a 2023-04-18 22:55:07,307 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/10ec4eceb5c7437ca78cb482a98a0630 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/10ec4eceb5c7437ca78cb482a98a0630 2023-04-18 22:55:07,310 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/01ec655a3be64bc2886702010dd49d4f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/e329576153d24d34a6b3ed96926e4891, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/7e53f3ee831344049cca95e9aded315e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/afba6b9c1a2c4bc5bec7643f7cab78c5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/eac0a6b72d954ac48f23ef330d1681b7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/470a294393d14f2c8a6e854fdbd19457, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/52a9a5ad7ff44025829e8e9d1ba81a3e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/5bfda20667d6409db95e5e01940e9929, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/54cf765d3dab4c5cb230c068468bf0ba, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/798fa9a66d184e28a2fe5735466a4f86, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/3442ea87a15b490cbb6cda78b9588d0b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/60f4338e5c9d4f0db62f5204589c6c2d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/94127e0d75f442bd85728934f6bc520b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/cbb51ea96ec34ab393007f8bbc3ea413, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/54f3af4c0e5045f6b4c4910e217ce1d5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/935dd618655d4d74862b8195b3b4ef9d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/e9540b06c2cf4deeb8a5a620e69eb4c1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/9b9b0c28361e4bd29f3feb6f75e2685d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/dacb2dbd785249ff9b7c8abd6b86d068, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/a4511808f017450aacebcf96663bb411, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/d9aa2c8c96144d8c9ea8866ec6388cb9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/8c2a9c092d5f455380af8421f95b1657, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/3ce970096b264f9f95c586b558c51af8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/83f6a1842bdc46f597db55acbafed72c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/5ead824a7dc54232bb772d0361a74657, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/6c2e57de49da4342be094d2354acf952, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/7776cf08abcb4936a999c1939d7c5c4c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/f505308b61ef4a44a5c3b75a8449c857, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/c1eea17a9bbf441f98ce2fc14ee2d91c] to archive 2023-04-18 22:55:07,311 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:55:07,312 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/01ec655a3be64bc2886702010dd49d4f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/01ec655a3be64bc2886702010dd49d4f 2023-04-18 22:55:07,313 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/e329576153d24d34a6b3ed96926e4891 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/e329576153d24d34a6b3ed96926e4891 2023-04-18 22:55:07,314 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/7e53f3ee831344049cca95e9aded315e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/7e53f3ee831344049cca95e9aded315e 2023-04-18 22:55:07,315 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/afba6b9c1a2c4bc5bec7643f7cab78c5 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/afba6b9c1a2c4bc5bec7643f7cab78c5 2023-04-18 22:55:07,316 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/eac0a6b72d954ac48f23ef330d1681b7 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/eac0a6b72d954ac48f23ef330d1681b7 2023-04-18 22:55:07,317 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/470a294393d14f2c8a6e854fdbd19457 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/470a294393d14f2c8a6e854fdbd19457 2023-04-18 22:55:07,318 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/52a9a5ad7ff44025829e8e9d1ba81a3e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/52a9a5ad7ff44025829e8e9d1ba81a3e 2023-04-18 22:55:07,318 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/5bfda20667d6409db95e5e01940e9929 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/5bfda20667d6409db95e5e01940e9929 2023-04-18 22:55:07,319 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/54cf765d3dab4c5cb230c068468bf0ba to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/54cf765d3dab4c5cb230c068468bf0ba 2023-04-18 22:55:07,320 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/798fa9a66d184e28a2fe5735466a4f86 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/798fa9a66d184e28a2fe5735466a4f86 2023-04-18 22:55:07,321 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/3442ea87a15b490cbb6cda78b9588d0b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/3442ea87a15b490cbb6cda78b9588d0b 2023-04-18 22:55:07,322 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/60f4338e5c9d4f0db62f5204589c6c2d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/60f4338e5c9d4f0db62f5204589c6c2d 2023-04-18 22:55:07,322 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/94127e0d75f442bd85728934f6bc520b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/94127e0d75f442bd85728934f6bc520b 2023-04-18 22:55:07,323 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/cbb51ea96ec34ab393007f8bbc3ea413 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/cbb51ea96ec34ab393007f8bbc3ea413 2023-04-18 22:55:07,324 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/54f3af4c0e5045f6b4c4910e217ce1d5 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/54f3af4c0e5045f6b4c4910e217ce1d5 2023-04-18 22:55:07,325 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/935dd618655d4d74862b8195b3b4ef9d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/935dd618655d4d74862b8195b3b4ef9d 2023-04-18 22:55:07,325 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/e9540b06c2cf4deeb8a5a620e69eb4c1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/e9540b06c2cf4deeb8a5a620e69eb4c1 2023-04-18 22:55:07,326 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/9b9b0c28361e4bd29f3feb6f75e2685d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/9b9b0c28361e4bd29f3feb6f75e2685d 2023-04-18 22:55:07,327 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/dacb2dbd785249ff9b7c8abd6b86d068 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/dacb2dbd785249ff9b7c8abd6b86d068 2023-04-18 22:55:07,328 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/a4511808f017450aacebcf96663bb411 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/a4511808f017450aacebcf96663bb411 2023-04-18 22:55:07,328 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/d9aa2c8c96144d8c9ea8866ec6388cb9 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/d9aa2c8c96144d8c9ea8866ec6388cb9 2023-04-18 22:55:07,329 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/8c2a9c092d5f455380af8421f95b1657 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/8c2a9c092d5f455380af8421f95b1657 2023-04-18 22:55:07,330 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/3ce970096b264f9f95c586b558c51af8 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/3ce970096b264f9f95c586b558c51af8 2023-04-18 22:55:07,330 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/83f6a1842bdc46f597db55acbafed72c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/83f6a1842bdc46f597db55acbafed72c 2023-04-18 22:55:07,331 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/5ead824a7dc54232bb772d0361a74657 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/5ead824a7dc54232bb772d0361a74657 2023-04-18 22:55:07,332 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/6c2e57de49da4342be094d2354acf952 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/6c2e57de49da4342be094d2354acf952 2023-04-18 22:55:07,332 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/7776cf08abcb4936a999c1939d7c5c4c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/7776cf08abcb4936a999c1939d7c5c4c 2023-04-18 22:55:07,333 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/f505308b61ef4a44a5c3b75a8449c857 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/f505308b61ef4a44a5c3b75a8449c857 2023-04-18 22:55:07,334 DEBUG [StoreCloser-TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/c1eea17a9bbf441f98ce2fc14ee2d91c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/c1eea17a9bbf441f98ce2fc14ee2d91c 2023-04-18 22:55:07,337 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/recovered.edits/571.seqid, newMaxSeqId=571, maxSeqId=1 2023-04-18 22:55:07,338 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa. 2023-04-18 22:55:07,338 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 94dcab632e7fc6a2d56c5605ce1488fa: 2023-04-18 22:55:07,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:55:07,340 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=94dcab632e7fc6a2d56c5605ce1488fa, regionState=CLOSED 2023-04-18 22:55:07,340 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858507340"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681858507340"}]},"ts":"1681858507340"} 2023-04-18 22:55:07,342 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-18 22:55:07,342 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 94dcab632e7fc6a2d56c5605ce1488fa, server=jenkins-hbase4.apache.org,35971,1681858424869 in 1.5120 sec 2023-04-18 22:55:07,343 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-18 22:55:07,343 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=94dcab632e7fc6a2d56c5605ce1488fa, UNASSIGN in 1.5160 sec 2023-04-18 22:55:07,344 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858507344"}]},"ts":"1681858507344"} 2023-04-18 22:55:07,345 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-18 22:55:07,347 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-18 22:55:07,348 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5270 sec 2023-04-18 22:55:07,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-18 22:55:07,928 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-18 22:55:07,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-18 22:55:07,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:07,931 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:07,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-18 22:55:07,931 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:07,936 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:55:07,938 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/recovered.edits] 2023-04-18 22:55:07,940 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8273138e4ecc43c59ea6addf93986728 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8273138e4ecc43c59ea6addf93986728 2023-04-18 22:55:07,941 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8cb230e4b348482b845c2521920d0473 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/8cb230e4b348482b845c2521920d0473 2023-04-18 22:55:07,942 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/901d440b7e3642d194c9c15d77cf2f73 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/901d440b7e3642d194c9c15d77cf2f73 2023-04-18 22:55:07,943 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/ea997b5deab548e7877f4235b40be7fa to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/A/ea997b5deab548e7877f4235b40be7fa 2023-04-18 22:55:07,945 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/26df935fe64d4b06a061d426495750d6 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/26df935fe64d4b06a061d426495750d6 2023-04-18 22:55:07,946 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/35a3fd1685b54fa4a6e222d5aec24c9b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/35a3fd1685b54fa4a6e222d5aec24c9b 2023-04-18 22:55:07,947 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/61d53d34cf3e4cb1b8d05fd324e3030c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/61d53d34cf3e4cb1b8d05fd324e3030c 2023-04-18 22:55:07,948 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/cda72bf22b6f49f3bac3b964bbb6c4ec to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/B/cda72bf22b6f49f3bac3b964bbb6c4ec 2023-04-18 22:55:07,950 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/1e60045075c0437888f452d0b7c4697f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/1e60045075c0437888f452d0b7c4697f 2023-04-18 22:55:07,951 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/222bf10e7a434564ac9803506c427546 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/222bf10e7a434564ac9803506c427546 2023-04-18 22:55:07,952 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/b4977edd32cb4c15bc33059e6baafe15 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/b4977edd32cb4c15bc33059e6baafe15 2023-04-18 22:55:07,953 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/efa5dff680274e8dbb69027375fbb3b8 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/C/efa5dff680274e8dbb69027375fbb3b8 2023-04-18 22:55:07,955 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/recovered.edits/571.seqid to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa/recovered.edits/571.seqid 2023-04-18 22:55:07,956 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/94dcab632e7fc6a2d56c5605ce1488fa 2023-04-18 22:55:07,956 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 22:55:07,958 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:07,959 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-18 22:55:07,961 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-18 22:55:07,961 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:07,961 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-18 22:55:07,962 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681858507962"}]},"ts":"9223372036854775807"} 2023-04-18 22:55:07,963 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-18 22:55:07,963 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 94dcab632e7fc6a2d56c5605ce1488fa, NAME => 'TestAcidGuarantees,,1681858482898.94dcab632e7fc6a2d56c5605ce1488fa.', STARTKEY => '', ENDKEY => ''}] 2023-04-18 22:55:07,963 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-18 22:55:07,963 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681858507963"}]},"ts":"9223372036854775807"} 2023-04-18 22:55:07,964 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-18 22:55:07,968 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:07,968 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 40 msec 2023-04-18 22:55:08,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-18 22:55:08,032 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-18 22:55:08,044 INFO [Listener at localhost/36651] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=229 (was 234), OpenFileDescriptor=566 (was 565) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=399 (was 409), ProcessCount=169 (was 169), AvailableMemoryMB=6254 (was 6286) 2023-04-18 22:55:08,056 INFO [Listener at localhost/36651] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229, OpenFileDescriptor=566, MaxFileDescriptor=60000, SystemLoadAverage=399, ProcessCount=169, AvailableMemoryMB=6254 2023-04-18 22:55:08,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 22:55:08,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 22:55:08,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:08,061 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-18 22:55:08,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-18 22:55:08,061 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-18 22:55:08,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-18 22:55:08,062 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:08,063 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c empty. 2023-04-18 22:55:08,063 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:08,063 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 22:55:08,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-18 22:55:08,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-18 22:55:08,470 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 22:55:08,472 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1a8c3530d711f58ef72306d7c2b1ec7c, NAME => 'TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp 2023-04-18 22:55:08,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-18 22:55:08,878 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:55:08,878 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 1a8c3530d711f58ef72306d7c2b1ec7c, disabling compactions & flushes 2023-04-18 22:55:08,878 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:08,878 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:08,878 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. after waiting 0 ms 2023-04-18 22:55:08,878 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:08,878 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:08,878 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:08,880 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-18 22:55:08,881 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858508881"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681858508881"}]},"ts":"1681858508881"} 2023-04-18 22:55:08,882 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-18 22:55:08,882 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-18 22:55:08,883 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858508882"}]},"ts":"1681858508882"} 2023-04-18 22:55:08,883 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-18 22:55:08,889 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a8c3530d711f58ef72306d7c2b1ec7c, ASSIGN}] 2023-04-18 22:55:08,891 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a8c3530d711f58ef72306d7c2b1ec7c, ASSIGN 2023-04-18 22:55:08,891 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a8c3530d711f58ef72306d7c2b1ec7c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35971,1681858424869; forceNewPlan=false, retain=false 2023-04-18 22:55:09,042 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=1a8c3530d711f58ef72306d7c2b1ec7c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:09,043 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858509042"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858509042"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858509042"}]},"ts":"1681858509042"} 2023-04-18 22:55:09,044 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:55:09,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-18 22:55:09,198 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:09,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1a8c3530d711f58ef72306d7c2b1ec7c, NAME => 'TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.', STARTKEY => '', ENDKEY => ''} 2023-04-18 22:55:09,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:09,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:55:09,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:09,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:09,200 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:09,201 DEBUG [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A 2023-04-18 22:55:09,201 DEBUG [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A 2023-04-18 22:55:09,201 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:55:09,202 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a8c3530d711f58ef72306d7c2b1ec7c columnFamilyName A 2023-04-18 22:55:09,202 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.HStore(310): Store=1a8c3530d711f58ef72306d7c2b1ec7c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:55:09,202 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:09,203 DEBUG [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B 2023-04-18 22:55:09,203 DEBUG [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B 2023-04-18 22:55:09,203 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:55:09,203 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a8c3530d711f58ef72306d7c2b1ec7c columnFamilyName B 2023-04-18 22:55:09,204 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.HStore(310): Store=1a8c3530d711f58ef72306d7c2b1ec7c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:55:09,204 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:09,205 DEBUG [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C 2023-04-18 22:55:09,205 DEBUG [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C 2023-04-18 22:55:09,205 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:55:09,205 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a8c3530d711f58ef72306d7c2b1ec7c columnFamilyName C 2023-04-18 22:55:09,205 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.HStore(310): Store=1a8c3530d711f58ef72306d7c2b1ec7c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:55:09,205 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:09,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:09,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:09,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 22:55:09,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:09,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 22:55:09,211 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1a8c3530d711f58ef72306d7c2b1ec7c; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67354839, jitterRate=0.0036653131246566772}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 22:55:09,211 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:09,211 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c., pid=37, masterSystemTime=1681858509195 2023-04-18 22:55:09,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:09,213 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:09,213 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=1a8c3530d711f58ef72306d7c2b1ec7c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:09,213 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858509213"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681858509213"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681858509213"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681858509213"}]},"ts":"1681858509213"} 2023-04-18 22:55:09,215 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-18 22:55:09,215 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 in 170 msec 2023-04-18 22:55:09,216 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-18 22:55:09,216 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a8c3530d711f58ef72306d7c2b1ec7c, ASSIGN in 326 msec 2023-04-18 22:55:09,217 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-18 22:55:09,217 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858509217"}]},"ts":"1681858509217"} 2023-04-18 22:55:09,218 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-18 22:55:09,220 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-18 22:55:09,221 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-04-18 22:55:10,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-18 22:55:10,166 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-18 22:55:10,167 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x0bb27fd4 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:10,172 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49b5bd0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:10,173 DEBUG [Listener at localhost/36651] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:10,175 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41186, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:10,176 DEBUG [Listener at localhost/36651] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-18 22:55:10,177 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35494, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-18 22:55:10,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 22:55:10,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 22:55:10,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:10,588 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-18 22:55:10,589 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 22:55:10,589 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-18 22:55:10,591 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-18 22:55:10,594 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a8c3530d711f58ef72306d7c2b1ec7c, REOPEN/MOVE}] 2023-04-18 22:55:10,594 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a8c3530d711f58ef72306d7c2b1ec7c, REOPEN/MOVE 2023-04-18 22:55:10,595 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1a8c3530d711f58ef72306d7c2b1ec7c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:10,595 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858510595"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858510595"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858510595"}]},"ts":"1681858510595"} 2023-04-18 22:55:10,596 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:55:10,747 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:10,748 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1a8c3530d711f58ef72306d7c2b1ec7c, disabling compactions & flushes 2023-04-18 22:55:10,748 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:10,748 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:10,748 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. after waiting 0 ms 2023-04-18 22:55:10,748 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:10,753 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-18 22:55:10,753 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:10,753 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:10,753 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 1a8c3530d711f58ef72306d7c2b1ec7c to self. 2023-04-18 22:55:10,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:10,755 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1a8c3530d711f58ef72306d7c2b1ec7c, regionState=CLOSED 2023-04-18 22:55:10,755 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858510755"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681858510755"}]},"ts":"1681858510755"} 2023-04-18 22:55:10,757 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-18 22:55:10,757 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 in 160 msec 2023-04-18 22:55:10,757 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a8c3530d711f58ef72306d7c2b1ec7c, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35971,1681858424869; forceNewPlan=false, retain=true 2023-04-18 22:55:10,908 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1a8c3530d711f58ef72306d7c2b1ec7c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:10,908 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858510908"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858510908"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858510908"}]},"ts":"1681858510908"} 2023-04-18 22:55:10,909 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:55:11,065 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:11,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1a8c3530d711f58ef72306d7c2b1ec7c, NAME => 'TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.', STARTKEY => '', ENDKEY => ''} 2023-04-18 22:55:11,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:11,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:55:11,066 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:11,067 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:11,067 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:11,068 DEBUG [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A 2023-04-18 22:55:11,068 DEBUG [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A 2023-04-18 22:55:11,069 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:55:11,069 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a8c3530d711f58ef72306d7c2b1ec7c columnFamilyName A 2023-04-18 22:55:11,070 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.HStore(310): Store=1a8c3530d711f58ef72306d7c2b1ec7c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:55:11,070 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:11,071 DEBUG [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B 2023-04-18 22:55:11,071 DEBUG [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B 2023-04-18 22:55:11,071 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:55:11,071 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a8c3530d711f58ef72306d7c2b1ec7c columnFamilyName B 2023-04-18 22:55:11,072 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.HStore(310): Store=1a8c3530d711f58ef72306d7c2b1ec7c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:55:11,072 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:11,073 DEBUG [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C 2023-04-18 22:55:11,073 DEBUG [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C 2023-04-18 22:55:11,073 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:55:11,073 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a8c3530d711f58ef72306d7c2b1ec7c columnFamilyName C 2023-04-18 22:55:11,073 INFO [StoreOpener-1a8c3530d711f58ef72306d7c2b1ec7c-1] regionserver.HStore(310): Store=1a8c3530d711f58ef72306d7c2b1ec7c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:55:11,073 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:11,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:11,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:11,076 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 22:55:11,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:11,078 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1a8c3530d711f58ef72306d7c2b1ec7c; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67905055, jitterRate=0.01186417043209076}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 22:55:11,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:11,078 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c., pid=42, masterSystemTime=1681858511061 2023-04-18 22:55:11,080 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:11,080 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:11,080 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=1a8c3530d711f58ef72306d7c2b1ec7c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,080 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858511080"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681858511080"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681858511080"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681858511080"}]},"ts":"1681858511080"} 2023-04-18 22:55:11,082 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-18 22:55:11,082 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 in 172 msec 2023-04-18 22:55:11,083 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-18 22:55:11,083 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a8c3530d711f58ef72306d7c2b1ec7c, REOPEN/MOVE in 488 msec 2023-04-18 22:55:11,084 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-18 22:55:11,084 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 492 msec 2023-04-18 22:55:11,086 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 907 msec 2023-04-18 22:55:11,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-18 22:55:11,087 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x445919fd to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:11,092 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c8d927d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:11,092 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x4d010713 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:11,096 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@419b69bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:11,096 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x69100738 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:11,101 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c6cac96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:11,101 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x6f603c27 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:11,105 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a0df34, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:11,105 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x7e01d5e5 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:11,108 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10ee5f06, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:11,109 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x21128196 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:11,113 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4366fcdd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:11,113 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e86d9 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:11,116 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a35a28d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:11,117 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x711ebaf2 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:11,122 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a1d4246, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:11,122 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x0a7503d4 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:11,126 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@eb8f142, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:11,126 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x7eae4191 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:11,129 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ce79633, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:11,138 DEBUG [hconnection-0x585d597-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:11,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:11,139 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41194, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:11,139 DEBUG [hconnection-0x50c36aff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:11,140 DEBUG [hconnection-0x7bcd1569-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:11,140 DEBUG [hconnection-0x192c0c8d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:11,140 DEBUG [hconnection-0x4fe62ced-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:11,141 DEBUG [hconnection-0x56aa37fc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:11,141 DEBUG [hconnection-0x15ffa306-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:11,141 DEBUG [hconnection-0x290545e1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:11,141 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41238, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:11,143 DEBUG [hconnection-0x41dc191f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:11,144 DEBUG [hconnection-0xcd4597-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:11,144 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41222, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:11,145 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41256, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:11,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:11,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:11,145 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41260, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:11,144 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41208, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:11,145 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:11,147 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:11,147 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:11,147 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:11,147 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41244, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:11,147 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41266, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:11,149 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:11,149 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:11,149 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41262, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:11,149 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41264, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:11,153 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:11,153 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:11,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:11,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:11,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:11,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:11,160 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:11,160 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:11,160 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,161 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:11,160 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:11,161 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:11,161 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:11,162 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:11,162 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:11,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:11,169 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:11,169 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a8c3530d711f58ef72306d7c2b1ec7c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-18 22:55:11,171 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=A 2023-04-18 22:55:11,171 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:11,171 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=B 2023-04-18 22:55:11,171 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:11,171 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=C 2023-04-18 22:55:11,171 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:11,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858571188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858571189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858571190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858571191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858571192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858571293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858571298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858571298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858571298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858571298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858571498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858571501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858571502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858571502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858571502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,590 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418e066f43cde524156a793516770fb0885 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e066f43cde524156a793516770fb0885 2023-04-18 22:55:11,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858571802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858571807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858571807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858571807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:11,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:11,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858571807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,001 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/989a18cff2ce4570b0b3cad90946b718 2023-04-18 22:55:12,012 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/55311bdfc65b49cda69823225840a342 2023-04-18 22:55:12,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:12,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858572310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:12,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858572311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:12,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858572313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:12,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858572314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:12,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858572315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,422 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/e5ed8b8a8d194287a6c64e3159ae6c37 2023-04-18 22:55:12,426 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/989a18cff2ce4570b0b3cad90946b718 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/989a18cff2ce4570b0b3cad90946b718 2023-04-18 22:55:12,429 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/989a18cff2ce4570b0b3cad90946b718, entries=150, sequenceid=18, filesize=25.1 K 2023-04-18 22:55:12,430 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/55311bdfc65b49cda69823225840a342 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/55311bdfc65b49cda69823225840a342 2023-04-18 22:55:12,434 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/55311bdfc65b49cda69823225840a342, entries=150, sequenceid=18, filesize=11.6 K 2023-04-18 22:55:12,435 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/e5ed8b8a8d194287a6c64e3159ae6c37 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/e5ed8b8a8d194287a6c64e3159ae6c37 2023-04-18 22:55:12,438 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/e5ed8b8a8d194287a6c64e3159ae6c37, entries=150, sequenceid=18, filesize=11.6 K 2023-04-18 22:55:12,439 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 1a8c3530d711f58ef72306d7c2b1ec7c in 1270ms, sequenceid=18, compaction requested=false 2023-04-18 22:55:12,439 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:12,439 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:12,439 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:12,439 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:12,439 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,439 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:12,439 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:12,441 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,441 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:12,441 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:12,441 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:12,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:12,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@278f6945[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:12,443 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:12,443 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:12,444 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:12,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:12,444 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:12,444 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:12,444 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:12,444 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,444 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:12,445 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:12,445 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:12,445 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,446 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:12,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:12,448 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:12,448 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:12,448 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:12,448 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:12,448 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:12,450 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:12,450 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:12,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:12,450 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:12,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:12,451 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,452 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,455 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,455 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:12,455 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:12,455 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:12,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:12,455 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:12,455 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:12,455 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:12,455 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:12,455 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:12,455 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:12,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:12,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:12,455 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:12,455 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:12,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:12,556 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:12,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:12,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:12,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:12,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:12,559 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:12,559 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:12,559 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:12,559 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:12,560 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:12,560 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:12,561 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:12,561 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:12,561 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:12,561 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:12,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:12,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:12,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:12,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:12,562 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:12,562 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:12,562 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:12,562 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:12,563 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:12,563 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:12,563 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:12,563 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a8c3530d711f58ef72306d7c2b1ec7c 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-18 22:55:12,563 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=A 2023-04-18 22:55:12,563 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:12,563 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=B 2023-04-18 22:55:12,563 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:12,563 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=C 2023-04-18 22:55:12,563 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:12,865 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-18 22:55:12,978 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418194d3e006c2f47e082203b30e7f0b78a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418194d3e006c2f47e082203b30e7f0b78a 2023-04-18 22:55:13,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:13,321 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:13,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858573330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858573330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858573331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858573331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858573332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,383 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/95b831c3145a45b69e4ea608cc507227 2023-04-18 22:55:13,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858573433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858573433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858573435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858573435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858573436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858573637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858573638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858573639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858573639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858573639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,805 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/1339686c5fe749c1bca3a3c4ad76f89a 2023-04-18 22:55:13,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858573939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858573940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858573943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858573944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:13,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:13,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858573945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,215 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/fc606345420d4f15ae74f14caf143a09 2023-04-18 22:55:14,219 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/95b831c3145a45b69e4ea608cc507227 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/95b831c3145a45b69e4ea608cc507227 2023-04-18 22:55:14,222 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/95b831c3145a45b69e4ea608cc507227, entries=150, sequenceid=36, filesize=25.1 K 2023-04-18 22:55:14,223 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/1339686c5fe749c1bca3a3c4ad76f89a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/1339686c5fe749c1bca3a3c4ad76f89a 2023-04-18 22:55:14,226 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/1339686c5fe749c1bca3a3c4ad76f89a, entries=150, sequenceid=36, filesize=11.6 K 2023-04-18 22:55:14,227 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/fc606345420d4f15ae74f14caf143a09 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/fc606345420d4f15ae74f14caf143a09 2023-04-18 22:55:14,232 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/fc606345420d4f15ae74f14caf143a09, entries=150, sequenceid=36, filesize=11.6 K 2023-04-18 22:55:14,233 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 1a8c3530d711f58ef72306d7c2b1ec7c in 1670ms, sequenceid=36, compaction requested=false 2023-04-18 22:55:14,233 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:14,233 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:14,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:14,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:14,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,234 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:14,234 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:14,235 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,235 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:14,236 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:14,236 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:14,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:14,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:14,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64f40635[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:14,237 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:14,237 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:14,239 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:14,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:14,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:14,239 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:14,239 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:14,239 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:14,239 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,239 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:14,240 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:14,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:14,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:14,240 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:14,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:14,242 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:14,242 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:14,242 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:14,242 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:14,242 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:14,245 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:14,245 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:14,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:14,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:14,245 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:14,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:14,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:14,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:14,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:14,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:14,245 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:14,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:14,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:14,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:14,247 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:14,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,249 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,249 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:14,249 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:14,249 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:14,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:14,249 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:14,249 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:14,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:14,249 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:14,249 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:14,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:14,249 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:14,249 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:14,249 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:14,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:14,250 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:14,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:14,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:14,349 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:14,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:14,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:14,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:14,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:14,353 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:14,353 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:14,353 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:14,353 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:14,354 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:14,354 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:14,356 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:14,356 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:14,356 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,356 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:14,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:14,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:14,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:14,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:14,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:14,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:14,357 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:14,358 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:14,358 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:14,358 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:14,358 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:14,358 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a8c3530d711f58ef72306d7c2b1ec7c 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 22:55:14,359 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=A 2023-04-18 22:55:14,359 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:14,359 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=B 2023-04-18 22:55:14,359 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:14,359 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=C 2023-04-18 22:55:14,359 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:14,372 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304180d9111c25c734c238a86e5ecce6d905e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304180d9111c25c734c238a86e5ecce6d905e 2023-04-18 22:55:14,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:14,445 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:14,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:14,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858574463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:14,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858574463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:14,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858574464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:14,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858574465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:14,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858574470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:14,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858574571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:14,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858574571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:14,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858574571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:14,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858574571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:14,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858574577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,778 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/03424cafea584e71ab3a50e220c8887d 2023-04-18 22:55:14,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:14,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858574776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:14,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858574776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:14,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858574777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:14,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858574777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:14,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858574782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:14,790 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/4e3a5d36b73549bd98e1b56c3fac56f9 2023-04-18 22:55:15,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858575082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858575082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858575084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858575084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858575084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,205 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/6f8a9b99158b4d08b2f0489acfd4f8c3 2023-04-18 22:55:15,214 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/03424cafea584e71ab3a50e220c8887d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/03424cafea584e71ab3a50e220c8887d 2023-04-18 22:55:15,218 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/03424cafea584e71ab3a50e220c8887d, entries=150, sequenceid=50, filesize=25.1 K 2023-04-18 22:55:15,219 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/4e3a5d36b73549bd98e1b56c3fac56f9 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/4e3a5d36b73549bd98e1b56c3fac56f9 2023-04-18 22:55:15,223 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/4e3a5d36b73549bd98e1b56c3fac56f9, entries=150, sequenceid=50, filesize=11.6 K 2023-04-18 22:55:15,224 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/6f8a9b99158b4d08b2f0489acfd4f8c3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/6f8a9b99158b4d08b2f0489acfd4f8c3 2023-04-18 22:55:15,227 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/6f8a9b99158b4d08b2f0489acfd4f8c3, entries=150, sequenceid=50, filesize=11.6 K 2023-04-18 22:55:15,228 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 1a8c3530d711f58ef72306d7c2b1ec7c in 870ms, sequenceid=50, compaction requested=true 2023-04-18 22:55:15,228 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:15,228 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:15,228 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:15,228 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:15,228 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,228 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:15,228 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:15,230 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:15,230 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:15,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:15,230 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:15,230 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:15,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:15,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:15,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:15,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:15,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:15,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ca9c056[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:15,232 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:15,232 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:15,237 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:15,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:15,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:15,238 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:15,238 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:15,238 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:15,238 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,238 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:15,239 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:15,239 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,239 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:15,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:15,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:15,239 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:15,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:15,242 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:15,242 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:15,242 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:15,242 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:15,242 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:15,246 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:15,246 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:15,247 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:15,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:15,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:15,247 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:15,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:15,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:15,248 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:15,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:15,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:15,258 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,258 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:15,258 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:15,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:15,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:15,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:15,258 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:15,258 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:15,259 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:15,259 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:15,259 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:15,259 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,259 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:15,259 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:15,259 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:15,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:15,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:15,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:15,359 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:15,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:15,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:15,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:15,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:15,363 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:15,363 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:15,363 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:15,363 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:15,366 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:15,366 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:15,368 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:15,368 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:15,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:15,369 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,369 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:15,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:15,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:15,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:15,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:15,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:15,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:15,370 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:15,370 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:15,370 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:15,370 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:15,370 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:15,370 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:15,371 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:15,371 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a8c3530d711f58ef72306d7c2b1ec7c 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-18 22:55:15,371 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=A 2023-04-18 22:55:15,371 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:15,371 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=B 2023-04-18 22:55:15,371 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:15,371 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=C 2023-04-18 22:55:15,371 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:15,395 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418f2798b13ae934d068fbcde573f3f4e4c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f2798b13ae934d068fbcde573f3f4e4c 2023-04-18 22:55:15,420 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=68, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/e0278fa4ee18486398674b96a95cb9df 2023-04-18 22:55:15,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:15,592 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:15,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858575603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858575604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858575605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858575606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858575607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858575711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858575711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858575713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858575713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858575713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,837 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/5185f412159943f0b08531fd97409490 2023-04-18 22:55:15,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858575918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858575918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858575918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858575918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:15,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:15,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858575918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:16,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858576221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:16,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858576221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:16,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858576221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:16,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858576222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:16,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858576222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,250 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/f2203046660249ac938441cf9d55f965 2023-04-18 22:55:16,254 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/e0278fa4ee18486398674b96a95cb9df as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/e0278fa4ee18486398674b96a95cb9df 2023-04-18 22:55:16,257 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/e0278fa4ee18486398674b96a95cb9df, entries=150, sequenceid=68, filesize=25.1 K 2023-04-18 22:55:16,258 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/5185f412159943f0b08531fd97409490 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5185f412159943f0b08531fd97409490 2023-04-18 22:55:16,261 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5185f412159943f0b08531fd97409490, entries=150, sequenceid=68, filesize=11.6 K 2023-04-18 22:55:16,262 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/f2203046660249ac938441cf9d55f965 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/f2203046660249ac938441cf9d55f965 2023-04-18 22:55:16,266 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/f2203046660249ac938441cf9d55f965, entries=150, sequenceid=68, filesize=11.6 K 2023-04-18 22:55:16,266 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 1a8c3530d711f58ef72306d7c2b1ec7c in 895ms, sequenceid=68, compaction requested=true 2023-04-18 22:55:16,266 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:16,266 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:16,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:16,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:16,267 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,267 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:16,267 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:16,268 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:16,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:16,268 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:16,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:16,269 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:16,269 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:16,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:16,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44d641ab[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:16,270 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:16,270 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:16,273 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:16,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:16,273 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:16,273 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:16,273 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:16,273 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,273 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:16,274 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,275 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:16,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:16,277 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:16,277 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:16,277 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:16,277 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:16,277 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:16,280 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:16,280 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:16,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:16,280 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:16,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:16,281 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,282 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,284 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,284 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:16,284 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:16,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:16,284 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:16,284 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:16,284 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,284 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:16,284 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:16,284 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:16,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:16,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:16,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:16,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:16,284 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:16,285 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:16,285 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:16,385 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:16,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:16,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:16,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:16,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:16,389 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:16,389 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:16,389 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:16,389 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:16,389 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:16,389 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:16,395 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:16,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:16,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:16,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:16,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:16,396 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,396 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:16,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:16,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:16,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:16,396 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:16,396 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:16,396 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:16,396 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:16,397 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:16,397 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:16,397 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:16,397 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a8c3530d711f58ef72306d7c2b1ec7c 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 22:55:16,397 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=A 2023-04-18 22:55:16,397 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:16,397 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=B 2023-04-18 22:55:16,397 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:16,397 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=C 2023-04-18 22:55:16,397 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:16,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:16,727 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:16,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:16,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:16,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858576746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858576747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:16,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858576775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:16,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858576775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:16,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858576775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,813 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418653268c60faa4c9f8ef3efe9026c87ef to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418653268c60faa4c9f8ef3efe9026c87ef 2023-04-18 22:55:16,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:16,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858576876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:16,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858576876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:16,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858576884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:16,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858576884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:16,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:16,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858576884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:17,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858577082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:17,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858577082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:17,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858577089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:17,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858577089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:17,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858577089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,219 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=82, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/1769f5ebfe084157938ddf411de17632 2023-04-18 22:55:17,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:17,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858577388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:17,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858577389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:17,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858577393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:17,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858577395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:17,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858577396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,629 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/ebd79e873d4642e2bbebb23b69dadf9a 2023-04-18 22:55:17,638 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/167ceab6b35a48f9a0984c0f6c9b91f6 2023-04-18 22:55:17,641 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/1769f5ebfe084157938ddf411de17632 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/1769f5ebfe084157938ddf411de17632 2023-04-18 22:55:17,645 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/1769f5ebfe084157938ddf411de17632, entries=150, sequenceid=82, filesize=25.1 K 2023-04-18 22:55:17,646 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/ebd79e873d4642e2bbebb23b69dadf9a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/ebd79e873d4642e2bbebb23b69dadf9a 2023-04-18 22:55:17,649 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/ebd79e873d4642e2bbebb23b69dadf9a, entries=150, sequenceid=82, filesize=11.6 K 2023-04-18 22:55:17,650 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/167ceab6b35a48f9a0984c0f6c9b91f6 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/167ceab6b35a48f9a0984c0f6c9b91f6 2023-04-18 22:55:17,653 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/167ceab6b35a48f9a0984c0f6c9b91f6, entries=150, sequenceid=82, filesize=11.6 K 2023-04-18 22:55:17,654 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=114.05 KB/116790 for 1a8c3530d711f58ef72306d7c2b1ec7c in 1257ms, sequenceid=82, compaction requested=true 2023-04-18 22:55:17,654 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:17,654 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:17,654 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:17,654 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:17,654 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,654 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:17,654 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:17,656 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,656 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:17,656 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:17,656 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:17,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:17,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:17,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:17,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@184fdd35[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:17,658 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:17,658 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:17,659 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:17,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:17,659 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:17,659 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,659 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:17,659 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:17,659 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:17,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:17,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:17,661 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:17,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:17,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:17,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:17,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:17,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:17,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:17,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:17,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:17,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:17,663 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:17,663 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:17,663 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:17,663 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:17,663 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:17,666 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:17,666 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:17,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:17,666 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:17,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:17,666 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:17,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:17,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:17,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:17,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:17,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,668 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:17,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:17,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,670 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:17,670 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:17,671 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:17,671 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:17,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:17,670 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:17,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:17,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:17,671 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:17,671 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,671 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:17,671 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:17,671 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:17,671 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:17,671 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:17,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:17,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:17,771 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:17,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:17,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:17,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:17,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:17,775 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:17,775 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:17,775 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:17,775 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:17,775 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:17,775 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:17,777 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,777 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:17,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:17,777 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,777 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:17,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:17,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:17,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:17,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:17,778 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:17,778 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:17,778 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:17,778 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:17,779 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:17,779 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:17,779 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:17,779 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a8c3530d711f58ef72306d7c2b1ec7c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 22:55:17,779 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=A 2023-04-18 22:55:17,779 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:17,779 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=B 2023-04-18 22:55:17,779 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:17,779 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=C 2023-04-18 22:55:17,779 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:17,795 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041800533a8e0e284903af9834116b73310d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041800533a8e0e284903af9834116b73310d 2023-04-18 22:55:17,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:17,902 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:17,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:17,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858577915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:17,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858577916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:17,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858577917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:17,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858577919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:17,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:17,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858577920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:18,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:18,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858578021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:18,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:18,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858578028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:18,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:18,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858578028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:18,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:18,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858578030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:18,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:18,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858578036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:18,200 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=102, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/42008a786af74cc6a51a8231afd33a81 2023-04-18 22:55:18,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:18,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858578228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:18,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:18,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858578235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:18,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:18,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858578236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:18,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:18,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858578236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:18,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:18,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858578244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:18,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:18,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858578533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:18,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:18,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858578542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:18,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:18,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858578544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:18,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:18,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858578544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:18,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:18,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858578551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:18,610 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/65ee99ddf4e8487c826f14099aca1b79 2023-04-18 22:55:19,020 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/d533e7066f8840e99f19c13ebfcc0791 2023-04-18 22:55:19,025 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/42008a786af74cc6a51a8231afd33a81 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/42008a786af74cc6a51a8231afd33a81 2023-04-18 22:55:19,028 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/42008a786af74cc6a51a8231afd33a81, entries=150, sequenceid=102, filesize=25.1 K 2023-04-18 22:55:19,029 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/65ee99ddf4e8487c826f14099aca1b79 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/65ee99ddf4e8487c826f14099aca1b79 2023-04-18 22:55:19,032 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/65ee99ddf4e8487c826f14099aca1b79, entries=150, sequenceid=102, filesize=11.6 K 2023-04-18 22:55:19,033 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/d533e7066f8840e99f19c13ebfcc0791 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/d533e7066f8840e99f19c13ebfcc0791 2023-04-18 22:55:19,036 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/d533e7066f8840e99f19c13ebfcc0791, entries=150, sequenceid=102, filesize=11.6 K 2023-04-18 22:55:19,037 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 1a8c3530d711f58ef72306d7c2b1ec7c in 1258ms, sequenceid=102, compaction requested=true 2023-04-18 22:55:19,037 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:19,037 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:19,037 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:19,037 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:19,037 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,038 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:19,038 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:19,040 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:19,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c4ea5a7[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:19,042 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:19,042 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,043 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,043 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:19,043 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:19,043 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:19,043 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,043 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:19,044 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:19,044 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:19,044 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,045 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:19,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:19,047 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a8c3530d711f58ef72306d7c2b1ec7c 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 22:55:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=A 2023-04-18 22:55:19,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:19,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=B 2023-04-18 22:55:19,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:19,048 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=C 2023-04-18 22:55:19,048 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:19,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:19,048 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:19,048 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:19,048 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:19,048 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:19,048 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:19,051 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,051 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:19,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,051 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,056 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,058 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,061 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:19,061 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:19,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:19,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:19,061 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:19,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:19,062 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:19,062 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:19,062 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,062 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:19,062 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,062 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,062 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,062 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,062 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:19,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:19,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858579093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858579095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858579094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858579095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858579096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,162 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:19,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:19,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:19,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:19,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:19,166 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:19,166 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:19,166 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:19,166 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:19,166 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,166 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,168 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:19,168 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:19,168 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,168 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:19,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:19,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:19,169 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:19,169 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:19,169 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:19,169 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:19,169 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:19,169 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:19,169 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:19,170 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:19,170 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:19,170 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:19,170 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:19,170 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,170 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:19,170 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:19,171 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,172 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,172 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,172 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:19,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e277508[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:19,174 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:19,174 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,175 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,175 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:19,175 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,175 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:19,175 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:19,175 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:19,178 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,178 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:19,178 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,178 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:19,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:19,180 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:19,180 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:19,180 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:19,180 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:19,180 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:19,183 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,183 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,183 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:19,183 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,184 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,187 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:19,187 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,187 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:19,187 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:19,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:19,187 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,187 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,187 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:19,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:19,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:19,187 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,188 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,188 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,188 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:19,188 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:19,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:19,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:19,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858579205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858579205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858579205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858579205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858579205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,288 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:19,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:19,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:19,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:19,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:19,291 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:19,291 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:19,291 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:19,291 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:19,292 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,292 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,293 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,293 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:19,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:19,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:19,293 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,293 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:19,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:19,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:19,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:19,294 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:19,294 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:19,294 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:19,294 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:19,295 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:19,295 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:19,295 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:19,295 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:19,295 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:19,295 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:19,295 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:19,295 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,295 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:19,295 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:19,297 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,297 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,298 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,298 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:19,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@688aa47e[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:19,299 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:19,299 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,300 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,300 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:19,300 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:19,300 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:19,302 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,302 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:19,303 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:19,303 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,303 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:19,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,303 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:19,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:19,306 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:19,306 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:19,306 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:19,306 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:19,306 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:19,307 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,308 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,308 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:19,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:19,308 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,310 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,313 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,313 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:19,313 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:19,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:19,313 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,313 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,313 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,313 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,313 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:19,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:19,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:19,313 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:19,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:19,313 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:19,313 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:19,313 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,414 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:19,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:19,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:19,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:19,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:19,417 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:19,417 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:19,417 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:19,417 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:19,417 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,417 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858579411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,419 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,419 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:19,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858579411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:19,419 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,419 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:19,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858579411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858579411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858579412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:19,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:19,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:19,420 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:19,420 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:19,421 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:19,421 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:19,421 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:19,421 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:19,421 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:19,421 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:19,421 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:19,421 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:19,421 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:19,421 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,421 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:19,421 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:19,423 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,423 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,423 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,423 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:19,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eca6a2f[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:19,424 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:19,424 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,425 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,426 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:19,426 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:19,426 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:19,426 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,426 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:19,428 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:19,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:19,430 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:19,431 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:19,431 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:19,431 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:19,431 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:19,432 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,432 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,432 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:19,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:19,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:19,433 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,434 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,436 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:19,436 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:19,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:19,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:19,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:19,436 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:19,436 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:19,436 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,436 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:19,436 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,436 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:19,436 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,436 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,437 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,437 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,474 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304186c9b5e4de5344e9287ec442a023fe85b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186c9b5e4de5344e9287ec442a023fe85b 2023-04-18 22:55:19,537 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:19,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:19,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:19,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:19,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:19,540 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:19,540 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:19,540 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:19,540 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:19,541 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,541 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,545 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:19,545 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:19,545 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,545 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:19,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:19,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:19,546 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:19,546 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:19,546 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:19,546 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:19,546 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:19,547 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:19,547 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:19,547 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:19,547 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:19,547 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:19,547 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:19,547 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,547 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:19,547 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:19,549 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,549 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,549 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,549 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:19,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@278095cc[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:19,550 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:19,550 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,551 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,552 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:19,552 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:19,552 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:19,553 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,553 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:19,554 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:19,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:19,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:19,556 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:19,556 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:19,556 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:19,556 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:19,557 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:19,558 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,558 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,558 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:19,559 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:19,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:19,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,560 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,563 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,563 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,563 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,563 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,563 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:19,563 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:19,563 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:19,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:19,563 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,563 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:19,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:19,563 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:19,563 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:19,564 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:19,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:19,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:19,664 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:19,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:19,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:19,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:19,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:19,667 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:19,668 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:19,668 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:19,668 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:19,668 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,668 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,669 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,669 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:19,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:19,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:19,670 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,670 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:19,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:19,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:19,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:19,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:19,671 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:19,671 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:19,671 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:19,671 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:19,671 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:19,671 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:19,671 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:19,671 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:19,671 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:19,671 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:19,671 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:19,671 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,672 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:19,672 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:19,673 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,673 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,673 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,673 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:19,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aa2b01[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:19,675 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:19,675 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,677 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,677 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:19,677 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:19,677 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:19,677 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,677 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:19,679 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:19,679 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:19,679 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,679 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:19,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:19,681 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:19,681 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:19,681 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:19,681 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:19,681 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:19,683 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,683 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,683 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:19,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,683 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,685 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,688 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:19,688 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,688 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:19,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:19,688 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,688 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,689 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,689 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,689 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,689 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:19,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:19,688 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:19,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:19,689 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:19,689 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:19,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:19,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858579721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858579721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858579721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858579722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:19,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858579722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,789 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:19,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:19,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:19,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:19,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:19,793 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:19,793 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:19,793 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:19,793 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:19,793 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,793 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,795 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,795 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:19,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:19,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:19,796 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,796 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:19,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:19,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:19,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:19,797 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:19,797 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:19,797 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:19,797 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:19,797 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:19,797 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:19,797 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:19,797 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:19,797 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:19,797 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:19,797 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:19,797 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:19,797 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,798 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:19,798 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:19,799 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,799 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,800 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,800 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:19,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:19,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fbec67e[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:19,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:19,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,802 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,802 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:19,803 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:19,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:19,803 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:19,804 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:19,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:19,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:19,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:19,807 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:19,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:19,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:19,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:19,807 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:19,809 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,810 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:19,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,810 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,810 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,812 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,813 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,813 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:19,814 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:19,814 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:19,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:19,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:19,814 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:19,814 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,814 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:19,814 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,814 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:19,814 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,814 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:19,814 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,881 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=116, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/69ca83b9391c4097bb7d99d73b211926 2023-04-18 22:55:19,914 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:19,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:19,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:19,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:19,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:19,918 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:19,918 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:19,918 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:19,918 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:19,918 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,918 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,919 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,919 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:19,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:19,920 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,920 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:19,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:19,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:19,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:19,921 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:19,921 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:19,921 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:19,921 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:19,921 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:19,921 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:19,921 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:19,922 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:19,922 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:19,922 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:19,927 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,927 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,927 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,927 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:19,929 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:19,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@627848b4[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:19,929 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,930 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,930 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,930 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:19,932 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:19,932 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,932 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,932 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:19,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:19,934 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:19,934 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:19,934 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:19,934 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:19,934 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:19,936 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,936 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:19,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,936 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:19,936 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,937 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:19,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:19,941 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,941 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:19,941 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:19,941 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:19,941 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:19,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:19,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:19,941 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:19,942 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:19,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:19,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:19,942 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,942 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:19,942 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:19,942 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:19,942 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,042 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:20,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:20,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:20,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:20,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:20,046 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:20,046 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:20,046 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:20,046 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:20,046 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,046 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,048 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:20,048 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:20,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:20,048 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,048 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:20,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:20,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:20,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:20,049 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:20,049 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:20,049 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:20,050 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:20,050 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:20,050 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:20,050 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:20,051 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,051 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:20,052 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,052 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:20,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:20,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41759fc2[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:20,053 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:20,053 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,055 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,055 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:20,055 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:20,055 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,055 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:20,055 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:20,057 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:20,057 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,057 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:20,057 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:20,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:20,060 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:20,060 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:20,060 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:20,060 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:20,060 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:20,061 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,062 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:20,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,062 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:20,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:20,062 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,064 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,066 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,066 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,066 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:20,066 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:20,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:20,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:20,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:20,067 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:20,066 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,067 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,067 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,066 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:20,067 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:20,067 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,067 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:20,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:20,167 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:20,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:20,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:20,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:20,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:20,171 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:20,171 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:20,171 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:20,171 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:20,171 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,171 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,173 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,173 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:20,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:20,173 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,173 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:20,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:20,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:20,174 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:20,174 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:20,174 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:20,174 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:20,174 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:20,174 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:20,175 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:20,175 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:20,175 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:20,175 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:20,175 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:20,175 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,175 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:20,175 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:20,178 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,178 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:20,178 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,178 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:20,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:20,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:20,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@125446ce[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:20,179 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:20,179 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,181 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,181 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:20,181 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:20,181 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,181 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:20,181 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:20,182 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:20,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:20,186 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:20,186 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:20,186 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:20,186 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:20,186 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:20,188 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,188 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:20,188 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:20,188 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:20,193 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,196 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:20,196 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:20,196 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:20,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:20,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:20,196 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:20,196 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,196 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,196 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,196 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,196 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:20,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:20,196 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:20,196 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:20,197 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:20,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858580231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:20,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858580231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:20,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858580232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:20,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858580234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:20,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858580234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,298 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:20,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:20,299 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/ae997fa971214d769d5ed920094d0f32 2023-04-18 22:55:20,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:20,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:20,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:20,302 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:20,302 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:20,302 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:20,302 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:20,302 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,302 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,305 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:20,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:20,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:20,305 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,306 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,306 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:20,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:20,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:20,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:20,307 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:20,307 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:20,307 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:20,307 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:20,307 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:20,307 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:20,307 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:20,307 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:20,307 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:20,307 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:20,307 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:20,307 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,307 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:20,307 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:20,308 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/55defdabcabc46f9be0bd4fc71d75a1b 2023-04-18 22:55:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,313 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,313 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:20,314 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/69ca83b9391c4097bb7d99d73b211926 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/69ca83b9391c4097bb7d99d73b211926 2023-04-18 22:55:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:20,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:20,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b10efcc[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:20,316 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:20,316 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,317 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/69ca83b9391c4097bb7d99d73b211926, entries=200, sequenceid=116, filesize=31.9 K 2023-04-18 22:55:20,318 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,318 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:20,318 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:20,318 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:20,318 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/ae997fa971214d769d5ed920094d0f32 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/ae997fa971214d769d5ed920094d0f32 2023-04-18 22:55:20,318 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,318 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:20,323 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/ae997fa971214d769d5ed920094d0f32, entries=150, sequenceid=116, filesize=11.6 K 2023-04-18 22:55:20,323 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:20,323 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:20,323 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:20,323 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/55defdabcabc46f9be0bd4fc71d75a1b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/55defdabcabc46f9be0bd4fc71d75a1b 2023-04-18 22:55:20,324 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:20,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:20,326 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:20,326 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:20,326 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:20,326 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:20,326 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:20,327 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/55defdabcabc46f9be0bd4fc71d75a1b, entries=150, sequenceid=116, filesize=11.6 K 2023-04-18 22:55:20,328 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,328 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:20,328 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:20,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,328 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 1a8c3530d711f58ef72306d7c2b1ec7c in 1281ms, sequenceid=116, compaction requested=true 2023-04-18 22:55:20,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:20,329 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:20,329 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:20,329 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:20,329 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-18 22:55:20,329 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-18 22:55:20,329 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,329 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:20,330 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,331 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83202 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-18 22:55:20,331 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 1a8c3530d711f58ef72306d7c2b1ec7c/B is initiating minor compaction (all files) 2023-04-18 22:55:20,332 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a8c3530d711f58ef72306d7c2b1ec7c/B in TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:20,332 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 187162 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-18 22:55:20,332 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/55311bdfc65b49cda69823225840a342, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/1339686c5fe749c1bca3a3c4ad76f89a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/4e3a5d36b73549bd98e1b56c3fac56f9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5185f412159943f0b08531fd97409490, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/ebd79e873d4642e2bbebb23b69dadf9a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/65ee99ddf4e8487c826f14099aca1b79, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/ae997fa971214d769d5ed920094d0f32] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp, totalSize=81.3 K 2023-04-18 22:55:20,332 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 1a8c3530d711f58ef72306d7c2b1ec7c/A is initiating minor compaction (all files) 2023-04-18 22:55:20,332 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a8c3530d711f58ef72306d7c2b1ec7c/A in TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:20,332 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/989a18cff2ce4570b0b3cad90946b718, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/95b831c3145a45b69e4ea608cc507227, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/03424cafea584e71ab3a50e220c8887d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/e0278fa4ee18486398674b96a95cb9df, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/1769f5ebfe084157938ddf411de17632, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/42008a786af74cc6a51a8231afd33a81, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/69ca83b9391c4097bb7d99d73b211926] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp, totalSize=182.8 K 2023-04-18 22:55:20,332 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 55311bdfc65b49cda69823225840a342, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1681858511164 2023-04-18 22:55:20,332 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 989a18cff2ce4570b0b3cad90946b718, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1681858511164 2023-04-18 22:55:20,333 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 1339686c5fe749c1bca3a3c4ad76f89a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681858511179 2023-04-18 22:55:20,337 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:20,337 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:20,337 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:20,337 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:20,338 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 95b831c3145a45b69e4ea608cc507227, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681858511179 2023-04-18 22:55:20,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:20,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:20,337 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:20,338 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,338 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,338 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 4e3a5d36b73549bd98e1b56c3fac56f9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681858513324 2023-04-18 22:55:20,338 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,338 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:20,338 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:20,338 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:20,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:20,338 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 03424cafea584e71ab3a50e220c8887d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681858513324 2023-04-18 22:55:20,338 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:20,339 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 5185f412159943f0b08531fd97409490, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681858514463 2023-04-18 22:55:20,339 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting e0278fa4ee18486398674b96a95cb9df, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681858514463 2023-04-18 22:55:20,339 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting ebd79e873d4642e2bbebb23b69dadf9a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681858515597 2023-04-18 22:55:20,344 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 1769f5ebfe084157938ddf411de17632, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681858515597 2023-04-18 22:55:20,344 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 65ee99ddf4e8487c826f14099aca1b79, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681858516745 2023-04-18 22:55:20,344 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 42008a786af74cc6a51a8231afd33a81, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681858516745 2023-04-18 22:55:20,345 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting ae997fa971214d769d5ed920094d0f32, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681858517914 2023-04-18 22:55:20,345 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 69ca83b9391c4097bb7d99d73b211926, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681858517914 2023-04-18 22:55:20,356 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a8c3530d711f58ef72306d7c2b1ec7c#B#compaction#338 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:20,363 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a8c3530d711f58ef72306d7c2b1ec7c#A#compaction#339 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:20,385 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/4d59b3d64bdc47bea5aa7cb2184de198 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/4d59b3d64bdc47bea5aa7cb2184de198 2023-04-18 22:55:20,388 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 1a8c3530d711f58ef72306d7c2b1ec7c/A of 1a8c3530d711f58ef72306d7c2b1ec7c into 4d59b3d64bdc47bea5aa7cb2184de198(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:20,389 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:20,389 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c., storeName=1a8c3530d711f58ef72306d7c2b1ec7c/A, priority=9, startTime=1681858520329; duration=0sec 2023-04-18 22:55:20,389 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:20,389 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-18 22:55:20,390 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83202 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-18 22:55:20,390 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 1a8c3530d711f58ef72306d7c2b1ec7c/C is initiating minor compaction (all files) 2023-04-18 22:55:20,391 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a8c3530d711f58ef72306d7c2b1ec7c/C in TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:20,391 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/e5ed8b8a8d194287a6c64e3159ae6c37, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/fc606345420d4f15ae74f14caf143a09, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/6f8a9b99158b4d08b2f0489acfd4f8c3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/f2203046660249ac938441cf9d55f965, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/167ceab6b35a48f9a0984c0f6c9b91f6, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/d533e7066f8840e99f19c13ebfcc0791, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/55defdabcabc46f9be0bd4fc71d75a1b] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp, totalSize=81.3 K 2023-04-18 22:55:20,391 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting e5ed8b8a8d194287a6c64e3159ae6c37, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1681858511164 2023-04-18 22:55:20,391 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting fc606345420d4f15ae74f14caf143a09, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681858511179 2023-04-18 22:55:20,392 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 6f8a9b99158b4d08b2f0489acfd4f8c3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681858513324 2023-04-18 22:55:20,392 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting f2203046660249ac938441cf9d55f965, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1681858514463 2023-04-18 22:55:20,392 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 167ceab6b35a48f9a0984c0f6c9b91f6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681858515597 2023-04-18 22:55:20,392 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting d533e7066f8840e99f19c13ebfcc0791, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681858516745 2023-04-18 22:55:20,392 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 55defdabcabc46f9be0bd4fc71d75a1b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681858517914 2023-04-18 22:55:20,411 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a8c3530d711f58ef72306d7c2b1ec7c#C#compaction#340 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:20,439 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:20,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:20,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:20,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:20,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:20,442 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:20,442 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:20,442 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:20,442 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:20,442 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,442 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,444 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,444 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:20,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:20,444 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:20,444 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:20,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:20,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:20,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:20,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:20,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:20,445 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:20,445 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:20,445 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:20,445 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:20,445 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:20,445 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:20,445 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:20,446 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a8c3530d711f58ef72306d7c2b1ec7c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 22:55:20,446 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=A 2023-04-18 22:55:20,446 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:20,446 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=B 2023-04-18 22:55:20,446 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:20,446 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=C 2023-04-18 22:55:20,446 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:20,774 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/546a77eb26004ce7bb4bd9fb8eea44bb as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/546a77eb26004ce7bb4bd9fb8eea44bb 2023-04-18 22:55:20,778 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 1a8c3530d711f58ef72306d7c2b1ec7c/B of 1a8c3530d711f58ef72306d7c2b1ec7c into 546a77eb26004ce7bb4bd9fb8eea44bb(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:20,778 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:20,778 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c., storeName=1a8c3530d711f58ef72306d7c2b1ec7c/B, priority=9, startTime=1681858520329; duration=0sec 2023-04-18 22:55:20,778 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:20,825 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/5537f333033544ae8a540670c0965287 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/5537f333033544ae8a540670c0965287 2023-04-18 22:55:20,829 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 1a8c3530d711f58ef72306d7c2b1ec7c/C of 1a8c3530d711f58ef72306d7c2b1ec7c into 5537f333033544ae8a540670c0965287(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:20,829 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:20,830 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c., storeName=1a8c3530d711f58ef72306d7c2b1ec7c/C, priority=9, startTime=1681858520329; duration=0sec 2023-04-18 22:55:20,830 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:20,867 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041810dd632060cd461a9ccebcdae35efbbe to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041810dd632060cd461a9ccebcdae35efbbe 2023-04-18 22:55:21,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:21,245 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:21,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858581252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858581253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858581254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858581255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858581256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,272 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=136, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/3f40d826294845ba9ab3de678b22b4df 2023-04-18 22:55:21,281 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/5468d372759d420f914b0d03ee875923 2023-04-18 22:55:21,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858581357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858581357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858581358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858581362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858581362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858581565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858581565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858581566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858581566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858581566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,694 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/4b17ec3b9a8d439e9bfd7a929a3b4a62 2023-04-18 22:55:21,698 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/3f40d826294845ba9ab3de678b22b4df as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/3f40d826294845ba9ab3de678b22b4df 2023-04-18 22:55:21,702 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/3f40d826294845ba9ab3de678b22b4df, entries=150, sequenceid=136, filesize=25.3 K 2023-04-18 22:55:21,702 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/5468d372759d420f914b0d03ee875923 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5468d372759d420f914b0d03ee875923 2023-04-18 22:55:21,706 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5468d372759d420f914b0d03ee875923, entries=150, sequenceid=136, filesize=11.8 K 2023-04-18 22:55:21,706 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/4b17ec3b9a8d439e9bfd7a929a3b4a62 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/4b17ec3b9a8d439e9bfd7a929a3b4a62 2023-04-18 22:55:21,710 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/4b17ec3b9a8d439e9bfd7a929a3b4a62, entries=150, sequenceid=136, filesize=11.8 K 2023-04-18 22:55:21,710 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 1a8c3530d711f58ef72306d7c2b1ec7c in 1264ms, sequenceid=136, compaction requested=false 2023-04-18 22:55:21,710 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:21,710 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:21,711 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:21,711 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:21,711 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,711 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:21,711 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:21,714 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,714 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:21,714 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:21,714 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:21,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b8d31f2[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:21,715 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:21,715 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:21,716 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:21,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:21,717 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:21,717 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:21,717 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:21,717 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,717 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:21,718 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:21,718 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:21,718 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,718 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:21,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:21,721 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:21,721 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:21,721 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:21,721 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:21,721 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:21,723 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:21,723 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:21,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:21,723 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:21,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:21,724 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,728 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,731 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:21,731 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:21,731 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:21,731 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:21,731 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:21,731 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,731 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:21,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:21,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:21,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:21,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:21,732 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:21,732 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:21,731 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:21,731 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:21,732 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:21,832 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:21,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:21,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:21,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:21,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:21,835 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:21,835 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:21,835 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:21,835 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:21,836 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:21,836 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:21,837 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,837 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:21,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:21,837 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,837 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:21,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:21,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:21,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:21,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:21,838 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:21,838 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:21,838 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:21,839 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:21,839 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:21,839 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:21,839 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:21,839 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a8c3530d711f58ef72306d7c2b1ec7c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 22:55:21,839 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=A 2023-04-18 22:55:21,839 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:21,839 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=B 2023-04-18 22:55:21,839 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:21,839 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=C 2023-04-18 22:55:21,839 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:21,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:21,875 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:21,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858581889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858581890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858581891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858581895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858581895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:21,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:21,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858581996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:22,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:22,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858581996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:22,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:22,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858581997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:22,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:22,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858582001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:22,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:22,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858582002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:22,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:22,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858582199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:22,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:22,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858582203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:22,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:22,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858582203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:22,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:22,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858582205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:22,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:22,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858582205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:22,258 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418ee4a5bbd954f4f2ba4575b9d03460f25 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ee4a5bbd954f4f2ba4575b9d03460f25 2023-04-18 22:55:22,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:22,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858582504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:22,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:22,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858582507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:22,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:22,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858582508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:22,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:22,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858582510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:22,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:22,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858582511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:22,664 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=151, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/94a0fb0487604e4b89e5af6ec310bddb 2023-04-18 22:55:23,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:23,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858583013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:23,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858583017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:23,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858583018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:23,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858583019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:23,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858583019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,078 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/49b956d9a915423b9ad8549b2663567b 2023-04-18 22:55:23,490 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/6b99e9b595d841be90e4f3d08a95423d 2023-04-18 22:55:23,494 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/94a0fb0487604e4b89e5af6ec310bddb as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/94a0fb0487604e4b89e5af6ec310bddb 2023-04-18 22:55:23,497 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/94a0fb0487604e4b89e5af6ec310bddb, entries=150, sequenceid=151, filesize=25.3 K 2023-04-18 22:55:23,498 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/49b956d9a915423b9ad8549b2663567b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/49b956d9a915423b9ad8549b2663567b 2023-04-18 22:55:23,501 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/49b956d9a915423b9ad8549b2663567b, entries=150, sequenceid=151, filesize=11.8 K 2023-04-18 22:55:23,502 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/6b99e9b595d841be90e4f3d08a95423d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/6b99e9b595d841be90e4f3d08a95423d 2023-04-18 22:55:23,505 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/6b99e9b595d841be90e4f3d08a95423d, entries=150, sequenceid=151, filesize=11.8 K 2023-04-18 22:55:23,506 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1a8c3530d711f58ef72306d7c2b1ec7c in 1667ms, sequenceid=151, compaction requested=true 2023-04-18 22:55:23,506 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:23,506 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:23,506 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:23,506 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:23,506 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,506 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:23,506 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:23,508 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:23,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:23,508 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:23,508 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:23,508 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:23,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:23,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:23,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16eb2ace[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:23,510 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:23,510 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:23,511 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:23,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:23,511 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:23,511 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:23,511 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:23,511 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,511 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:23,513 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:23,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:23,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:23,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:23,515 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:23,516 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:23,516 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:23,516 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:23,516 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:23,517 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:23,517 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:23,517 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:23,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:23,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:23,518 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,521 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,524 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:23,524 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:23,524 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:23,525 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:23,524 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:23,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:23,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:23,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:23,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:23,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:23,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:23,525 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:23,525 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:23,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:23,625 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:23,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:23,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:23,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:23,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:23,629 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:23,629 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:23,629 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:23,629 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:23,629 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:23,630 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:23,631 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:23,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:23,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:23,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:23,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:23,632 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:23,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:23,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:23,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:23,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:23,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:23,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:23,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:23,633 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:23,633 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:23,633 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a8c3530d711f58ef72306d7c2b1ec7c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 22:55:23,633 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=A 2023-04-18 22:55:23,633 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:23,633 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=B 2023-04-18 22:55:23,633 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:23,633 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=C 2023-04-18 22:55:23,633 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:24,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:24,029 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:24,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858584039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858584040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858584040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858584045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858584046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,056 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418bcb24edc328d4ae5ad6ade0d12d0e966 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418bcb24edc328d4ae5ad6ade0d12d0e966 2023-04-18 22:55:24,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858584147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858584147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858584147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858584151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858584153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858584352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858584352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858584353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858584357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858584358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,462 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=170, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/23aff79838cd4625878921e7b4f97315 2023-04-18 22:55:24,473 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/56969893b1a14da78fad0c0e4660a5cc 2023-04-18 22:55:24,487 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/2db4b2eda1164545aec6d4255657b8d4 2023-04-18 22:55:24,491 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/23aff79838cd4625878921e7b4f97315 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/23aff79838cd4625878921e7b4f97315 2023-04-18 22:55:24,494 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/23aff79838cd4625878921e7b4f97315, entries=150, sequenceid=170, filesize=25.3 K 2023-04-18 22:55:24,495 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/56969893b1a14da78fad0c0e4660a5cc as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/56969893b1a14da78fad0c0e4660a5cc 2023-04-18 22:55:24,498 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/56969893b1a14da78fad0c0e4660a5cc, entries=150, sequenceid=170, filesize=11.8 K 2023-04-18 22:55:24,499 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/2db4b2eda1164545aec6d4255657b8d4 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/2db4b2eda1164545aec6d4255657b8d4 2023-04-18 22:55:24,502 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/2db4b2eda1164545aec6d4255657b8d4, entries=150, sequenceid=170, filesize=11.8 K 2023-04-18 22:55:24,503 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 1a8c3530d711f58ef72306d7c2b1ec7c in 870ms, sequenceid=170, compaction requested=true 2023-04-18 22:55:24,503 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:24,503 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:24,503 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:24,503 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:24,503 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,503 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:24,503 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:24,506 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,506 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:24,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:24,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:24,506 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:24,507 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:24,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:24,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:24,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26b97ceb[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:24,508 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:24,509 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:24,510 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:24,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:24,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:24,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:24,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:24,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:24,510 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,510 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:24,511 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,511 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:24,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:24,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:24,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:24,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:24,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:24,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:24,515 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:24,515 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:24,515 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:24,516 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:24,516 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:24,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:24,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:24,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:24,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:24,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:24,521 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,522 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:24,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:24,526 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:24,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:24,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:24,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:24,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,527 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:24,527 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:24,526 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:24,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:24,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:24,527 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:24,527 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:24,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:24,627 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:24,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:24,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:24,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:24,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:24,630 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:24,630 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:24,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:24,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:24,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:24,631 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:24,632 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:24,632 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:24,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:24,632 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,632 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:24,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:24,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:24,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:24,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:24,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:24,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:24,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:24,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:24,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:24,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:24,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:24,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:24,634 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:24,634 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:24,634 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a8c3530d711f58ef72306d7c2b1ec7c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 22:55:24,634 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=A 2023-04-18 22:55:24,634 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:24,634 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=B 2023-04-18 22:55:24,634 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:24,634 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=C 2023-04-18 22:55:24,634 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:24,646 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418b76aaf8d05a5483682f275164bd61b7a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418b76aaf8d05a5483682f275164bd61b7a 2023-04-18 22:55:24,652 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=183, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/a9a9a5e595a44e169f82fec3b0d30950 2023-04-18 22:55:24,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:24,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:24,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858584683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858584683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858584683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858584689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858584690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858584791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858584791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858584792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858584792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:24,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:24,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858584798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858584999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858585000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858585000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858585000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858585002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,061 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/5c7e1f3386f14fbaaa3228cea7922749 2023-04-18 22:55:25,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858585310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858585310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858585311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858585312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41264 deadline: 1681858585312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,471 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/f8415ae961bb44538fe65bbac798f356 2023-04-18 22:55:25,475 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/a9a9a5e595a44e169f82fec3b0d30950 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/a9a9a5e595a44e169f82fec3b0d30950 2023-04-18 22:55:25,479 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/a9a9a5e595a44e169f82fec3b0d30950, entries=150, sequenceid=183, filesize=25.3 K 2023-04-18 22:55:25,479 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/5c7e1f3386f14fbaaa3228cea7922749 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5c7e1f3386f14fbaaa3228cea7922749 2023-04-18 22:55:25,482 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5c7e1f3386f14fbaaa3228cea7922749, entries=150, sequenceid=183, filesize=11.8 K 2023-04-18 22:55:25,483 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/f8415ae961bb44538fe65bbac798f356 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/f8415ae961bb44538fe65bbac798f356 2023-04-18 22:55:25,488 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/f8415ae961bb44538fe65bbac798f356, entries=150, sequenceid=183, filesize=11.8 K 2023-04-18 22:55:25,488 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1a8c3530d711f58ef72306d7c2b1ec7c in 854ms, sequenceid=183, compaction requested=true 2023-04-18 22:55:25,488 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:25,488 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:25,488 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:25,488 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:25,488 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:25,490 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,490 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:25,491 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:25,491 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:25,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:25,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a0b997f[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:25,492 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:25,492 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:25,494 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:25,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:25,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,495 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:25,495 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:25,496 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:25,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:25,498 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:25,498 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:25,498 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:25,498 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:25,498 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:25,500 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:25,500 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:25,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:25,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:25,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:25,500 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:25,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:25,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:25,500 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:25,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:25,502 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:25,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:25,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:25,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:25,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:25,504 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:25,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:25,504 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:25,504 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:25,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:25,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:25,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:25,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:25,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:25,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:25,604 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:25,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:25,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:25,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:25,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:25,608 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:25,608 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:25,608 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:25,608 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:25,608 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:25,608 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:25,611 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:25,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:25,611 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:25,611 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:25,611 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:25,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:25,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:25,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:25,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:25,613 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:25,613 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:25,613 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a8c3530d711f58ef72306d7c2b1ec7c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 22:55:25,613 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=A 2023-04-18 22:55:25,613 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:25,613 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=B 2023-04-18 22:55:25,613 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:25,613 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=C 2023-04-18 22:55:25,613 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:25,628 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418f0a122e53e5e4a5ba718c94f20a27234 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f0a122e53e5e4a5ba718c94f20a27234 2023-04-18 22:55:25,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:25,818 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:25,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41194 deadline: 1681858585825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858585833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858585833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41260 deadline: 1681858585834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41264 deadline: 1681858585835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41194 deadline: 1681858585936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41256 deadline: 1681858585944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41244 deadline: 1681858585944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41260 deadline: 1681858585944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:25,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:25,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41264 deadline: 1681858585944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,033 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=202, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/ac22920fea7c46b9adc4609bb575ac1e 2023-04-18 22:55:26,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:26,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41194 deadline: 1681858586144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:26,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41260 deadline: 1681858586152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:26,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41264 deadline: 1681858586152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:26,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41256 deadline: 1681858586153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:26,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41244 deadline: 1681858586153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,443 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/765f294c99dc47039a099faadff83f61 2023-04-18 22:55:26,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:26,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41194 deadline: 1681858586452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:26,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41260 deadline: 1681858586460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:26,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41264 deadline: 1681858586460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:26,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41256 deadline: 1681858586461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:26,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41244 deadline: 1681858586462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,854 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/0fe297fe7758487f9b136a587aabbf1d 2023-04-18 22:55:26,858 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/ac22920fea7c46b9adc4609bb575ac1e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/ac22920fea7c46b9adc4609bb575ac1e 2023-04-18 22:55:26,862 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/ac22920fea7c46b9adc4609bb575ac1e, entries=150, sequenceid=202, filesize=25.3 K 2023-04-18 22:55:26,862 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/765f294c99dc47039a099faadff83f61 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/765f294c99dc47039a099faadff83f61 2023-04-18 22:55:26,866 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/765f294c99dc47039a099faadff83f61, entries=150, sequenceid=202, filesize=11.8 K 2023-04-18 22:55:26,867 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/0fe297fe7758487f9b136a587aabbf1d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/0fe297fe7758487f9b136a587aabbf1d 2023-04-18 22:55:26,873 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/0fe297fe7758487f9b136a587aabbf1d, entries=150, sequenceid=202, filesize=11.8 K 2023-04-18 22:55:26,874 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 1a8c3530d711f58ef72306d7c2b1ec7c in 1261ms, sequenceid=202, compaction requested=true 2023-04-18 22:55:26,874 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:26,874 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:26,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:26,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:26,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:26,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:26,876 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,876 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:26,876 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:26,876 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:26,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:26,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@689b256d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:26,879 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:26,879 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:26,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:26,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:26,881 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:26,881 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:26,881 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:26,881 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,881 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:26,883 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:26,883 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:26,883 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:26,883 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:26,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:26,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:26,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:26,886 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:26,886 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:26,886 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:26,886 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:26,886 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:26,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:26,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:26,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:26,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:26,890 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,892 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:26,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,894 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,894 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:26,894 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:26,894 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:26,894 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:26,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:26,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:26,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:26,895 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:26,895 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:26,895 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:26,895 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:26,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:26,895 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:26,895 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:26,895 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:26,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:26,962 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1a8c3530d711f58ef72306d7c2b1ec7c 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 22:55:26,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=A 2023-04-18 22:55:26,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:26,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=B 2023-04-18 22:55:26,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:26,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=C 2023-04-18 22:55:26,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:26,995 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:26,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:26,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:26,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:26,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:26,999 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:26,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:26,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:26,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:26,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:26,999 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,001 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:27,001 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:27,001 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,001 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:27,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41264 deadline: 1681858587000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:27,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:27,002 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:27,002 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:27,002 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:27,003 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:27,003 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:27,003 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:27,003 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:27,005 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,005 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,005 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,005 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:27,006 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:27,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2769cfea[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:27,006 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,008 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:27,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:27,009 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,009 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:27,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:27,010 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:27,010 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:27,010 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,010 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41244 deadline: 1681858587001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41256 deadline: 1681858587002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41194 deadline: 1681858587002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41260 deadline: 1681858587002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:27,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:27,014 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:27,014 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:27,014 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:27,014 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:27,014 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:27,016 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,016 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,016 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:27,016 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,018 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,026 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:27,026 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,026 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,026 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,026 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,026 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:27,026 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:27,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:27,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:27,026 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:27,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:27,026 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:27,027 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,027 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:27,027 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:27,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:27,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41264 deadline: 1681858587103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41244 deadline: 1681858587112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,127 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:27,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:27,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41256 deadline: 1681858587113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41194 deadline: 1681858587113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41260 deadline: 1681858587113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:27,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:27,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:27,130 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:27,130 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:27,130 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:27,130 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:27,131 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,131 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,132 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,132 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:27,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:27,132 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,132 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:27,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:27,133 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:27,133 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:27,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:27,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:27,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:27,134 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:27,134 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:27,134 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:27,134 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:27,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:27,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:27,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:27,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:27,136 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,136 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,136 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,136 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:27,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4adab1c8[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:27,137 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:27,137 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,138 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,139 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:27,139 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,139 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:27,139 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:27,139 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:27,141 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,141 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,141 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:27,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:27,143 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:27,144 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:27,144 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:27,144 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:27,144 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:27,145 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,145 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,145 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:27,146 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,147 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,149 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:27,149 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:27,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:27,149 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:27,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:27,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:27,149 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:27,149 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,149 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,149 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,150 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,150 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,150 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,150 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:27,150 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:27,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:27,250 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:27,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:27,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:27,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:27,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:27,254 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:27,254 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:27,254 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:27,254 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:27,255 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,255 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,256 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,256 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:27,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:27,257 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,257 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:27,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:27,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:27,258 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:27,258 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:27,258 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:27,258 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,259 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:27,259 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,266 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:27,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cbc098d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:27,275 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:27,275 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,277 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:27,277 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:27,277 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:27,277 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,277 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:27,279 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,279 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,279 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:27,279 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:27,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:27,282 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:27,282 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:27,282 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:27,282 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:27,282 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:27,284 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,284 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:27,284 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:27,284 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,287 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,291 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,291 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:27,291 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:27,291 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:27,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:27,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:27,291 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:27,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:27,291 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:27,291 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:27,291 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:27,291 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,292 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:27,292 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,292 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41264 deadline: 1681858587307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41244 deadline: 1681858587315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41256 deadline: 1681858587329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41194 deadline: 1681858587330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41260 deadline: 1681858587330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,374 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418bba62ceff7334e9cb443d55b39508644 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418bba62ceff7334e9cb443d55b39508644 2023-04-18 22:55:27,380 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=216, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/8beb23785ff442fca480387cf5483c2c 2023-04-18 22:55:27,392 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:27,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:27,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:27,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:27,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:27,395 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:27,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:27,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:27,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:27,396 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,396 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,397 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,397 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:27,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:27,399 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,399 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:27,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:27,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:27,404 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:27,404 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:27,404 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:27,404 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:27,404 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:27,404 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:27,404 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:27,404 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:27,404 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:27,406 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,406 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,407 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,407 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:27,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dbdf93a[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:27,408 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:27,408 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,409 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,410 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:27,410 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:27,410 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:27,410 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,410 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:27,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:27,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:27,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:27,420 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:27,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:27,424 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:27,425 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:27,425 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:27,425 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:27,425 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:27,427 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,427 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,427 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:27,428 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,430 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,435 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:27,435 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:27,435 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:27,435 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:27,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:27,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:27,435 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,435 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:27,435 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,436 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:27,436 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,436 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:27,436 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,436 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,536 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:27,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:27,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:27,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:27,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:27,539 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:27,539 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:27,539 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:27,539 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:27,539 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,539 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,542 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,542 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:27,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:27,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:27,542 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,542 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:27,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:27,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:27,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:27,543 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:27,543 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:27,543 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:27,543 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:27,543 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:27,543 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:27,543 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:27,543 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:27,544 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:27,544 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:27,544 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:27,544 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,544 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:27,544 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:27,545 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,545 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,545 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,545 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:27,547 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:27,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34c26bf8[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:27,547 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,549 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:27,549 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,549 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:27,550 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,550 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:27,550 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:27,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:27,553 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:27,553 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:27,553 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:27,553 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:27,553 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:27,555 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,555 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,556 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:27,556 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,557 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,559 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:27,559 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:27,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:27,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:27,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:27,560 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:27,560 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:27,559 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:27,559 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:27,559 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,561 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,561 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,561 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,561 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,561 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:27,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41264 deadline: 1681858587621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41244 deadline: 1681858587622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41260 deadline: 1681858587633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41194 deadline: 1681858587634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:27,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41256 deadline: 1681858587635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,660 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:27,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:27,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:27,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:27,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:27,669 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:27,669 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:27,669 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:27,669 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:27,669 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,669 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,671 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:27,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:27,671 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,671 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,671 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:27,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:27,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:27,674 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:27,674 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:27,674 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:27,674 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:27,676 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,676 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,677 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,677 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b89d859[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:27,678 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:27,678 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,682 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,682 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:27,682 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:27,682 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,682 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:27,682 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:27,684 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:27,684 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,684 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,684 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:27,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:27,687 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:27,687 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:27,687 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:27,687 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:27,687 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:27,689 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,689 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:27,689 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,689 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,691 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,693 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:27,693 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:27,693 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:27,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:27,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:27,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:27,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:27,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:27,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:27,693 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,694 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,694 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,694 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:27,694 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:27,694 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:27,694 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,695 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,695 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,791 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/261a63ee3bb74c2a9f1326f817e241a3 2023-04-18 22:55:27,795 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:27,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:27,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:27,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:27,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:27,799 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:27,799 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:27,799 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:27,799 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:27,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,805 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,805 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:27,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:27,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:27,805 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,805 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:27,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:27,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:27,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:27,808 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:27,808 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:27,808 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:27,808 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:27,808 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:27,808 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:27,808 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:27,808 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:27,808 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:27,809 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:27,809 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:27,809 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,809 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:27,809 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:27,810 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,810 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,811 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,811 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27f55cc4[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:27,812 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:27,812 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,813 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:27,814 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,814 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:27,814 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:27,815 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,815 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,815 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:27,815 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:27,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:27,818 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:27,818 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:27,818 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:27,818 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:27,818 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:27,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:27,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:27,821 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,822 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,825 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:27,825 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:27,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:27,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:27,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:27,825 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:27,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:27,826 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:27,826 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:27,826 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,827 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,827 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,827 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,827 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,827 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,827 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,926 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:27,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:27,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:27,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:27,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:27,932 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:27,932 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:27,932 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:27,932 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:27,933 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,933 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,934 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:27,934 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:27,934 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,935 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:27,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:27,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:27,936 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:27,936 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:27,936 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:27,936 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:27,936 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:27,936 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:27,936 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:27,936 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:27,937 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:27,937 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:27,937 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:27,937 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,937 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:27,937 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:27,944 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,944 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,944 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,944 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:27,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15b30529[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:27,948 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:27,948 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,949 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:27,949 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,949 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:27,951 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,951 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,951 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:27,951 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:27,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:27,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:27,954 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:27,954 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:27,954 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:27,954 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:27,954 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:27,955 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,955 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,955 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:27,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:27,956 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:27,957 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,961 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:27,961 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:27,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:27,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:27,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:27,961 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,961 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,961 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:27,961 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:27,961 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:27,961 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:27,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:27,962 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:27,961 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:27,962 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:27,962 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,062 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:28,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:28,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:28,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:28,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:28,070 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:28,070 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:28,070 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:28,070 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:28,070 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,070 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:28,072 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:28,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:28,072 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:28,073 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,073 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:28,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:28,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:28,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:28,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:28,074 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:28,074 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:28,074 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:28,074 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:28,075 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:28,075 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:28,075 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,075 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:28,075 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:28,078 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:28,078 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:28,079 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:28,079 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:28,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:28,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@328931cc[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:28,081 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:28,081 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:28,082 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:28,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:28,083 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:28,083 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:28,083 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:28,083 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,083 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:28,085 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:28,085 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:28,085 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:28,085 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:28,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:28,090 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:28,090 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:28,090 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:28,090 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:28,090 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:28,091 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,091 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:28,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,091 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:28,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:28,092 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,093 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,095 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:28,095 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:28,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:28,095 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:28,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:28,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:28,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:28,095 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,096 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:28,096 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:28,096 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,096 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:28,096 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:28,096 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:28,096 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:28,096 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:28,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:28,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41244 deadline: 1681858588129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:28,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41264 deadline: 1681858588132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:28,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41260 deadline: 1681858588143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:28,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41256 deadline: 1681858588147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:28,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41194 deadline: 1681858588148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,196 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:28,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:28,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:28,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:28,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:28,199 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:28,199 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:28,199 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:28,199 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:28,200 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,200 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:28,201 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,201 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:28,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:28,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:28,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:28,202 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,202 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:28,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:28,203 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:28,203 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:28,203 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:28,203 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:28,203 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:28,203 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:28,203 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:28,203 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:28,203 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:28,203 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:28,203 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:28,203 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,203 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:28,203 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:28,205 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/65bdfd86d5c244d6955622a0a6ea6643 2023-04-18 22:55:28,206 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,206 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:28,206 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:28,206 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:28,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:28,209 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:28,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57277cd4[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:28,209 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:28,209 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/8beb23785ff442fca480387cf5483c2c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/8beb23785ff442fca480387cf5483c2c 2023-04-18 22:55:28,210 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:28,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:28,211 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:28,211 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:28,211 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:28,211 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,211 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:28,213 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:28,214 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:28,214 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:28,214 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:28,215 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/8beb23785ff442fca480387cf5483c2c, entries=200, sequenceid=216, filesize=32.1 K 2023-04-18 22:55:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:28,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:28,215 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:28,216 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:28,216 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/261a63ee3bb74c2a9f1326f817e241a3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/261a63ee3bb74c2a9f1326f817e241a3 2023-04-18 22:55:28,216 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:28,216 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:28,216 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:28,218 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,218 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:28,218 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,218 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:28,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:28,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:28,220 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:28,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:28,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/261a63ee3bb74c2a9f1326f817e241a3, entries=150, sequenceid=216, filesize=11.8 K 2023-04-18 22:55:28,223 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/65bdfd86d5c244d6955622a0a6ea6643 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/65bdfd86d5c244d6955622a0a6ea6643 2023-04-18 22:55:28,224 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,224 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:28,224 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:28,224 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:28,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:28,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:28,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:28,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:28,224 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:28,224 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:28,225 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,225 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:28,225 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:28,225 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,225 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:28,225 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:28,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:28,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:28,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:28,227 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/65bdfd86d5c244d6955622a0a6ea6643, entries=150, sequenceid=216, filesize=11.8 K 2023-04-18 22:55:28,228 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=114.05 KB/116790 for 1a8c3530d711f58ef72306d7c2b1ec7c in 1266ms, sequenceid=216, compaction requested=true 2023-04-18 22:55:28,228 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:28,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:28,229 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-18 22:55:28,229 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-18 22:55:28,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:28,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:28,231 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84342 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-18 22:55:28,231 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 1a8c3530d711f58ef72306d7c2b1ec7c/B is initiating minor compaction (all files) 2023-04-18 22:55:28,231 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a8c3530d711f58ef72306d7c2b1ec7c/B in TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:28,231 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/546a77eb26004ce7bb4bd9fb8eea44bb, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5468d372759d420f914b0d03ee875923, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/49b956d9a915423b9ad8549b2663567b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/56969893b1a14da78fad0c0e4660a5cc, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5c7e1f3386f14fbaaa3228cea7922749, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/765f294c99dc47039a099faadff83f61, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/261a63ee3bb74c2a9f1326f817e241a3] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp, totalSize=82.4 K 2023-04-18 22:55:28,231 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 188352 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-18 22:55:28,231 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 1a8c3530d711f58ef72306d7c2b1ec7c/A is initiating minor compaction (all files) 2023-04-18 22:55:28,231 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 546a77eb26004ce7bb4bd9fb8eea44bb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681858517914 2023-04-18 22:55:28,231 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a8c3530d711f58ef72306d7c2b1ec7c/A in TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:28,232 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/4d59b3d64bdc47bea5aa7cb2184de198, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/3f40d826294845ba9ab3de678b22b4df, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/94a0fb0487604e4b89e5af6ec310bddb, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/23aff79838cd4625878921e7b4f97315, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/a9a9a5e595a44e169f82fec3b0d30950, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/ac22920fea7c46b9adc4609bb575ac1e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/8beb23785ff442fca480387cf5483c2c] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp, totalSize=183.9 K 2023-04-18 22:55:28,232 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 5468d372759d420f914b0d03ee875923, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681858519092 2023-04-18 22:55:28,232 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 4d59b3d64bdc47bea5aa7cb2184de198, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681858517914 2023-04-18 22:55:28,232 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 49b956d9a915423b9ad8549b2663567b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681858521252 2023-04-18 22:55:28,232 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 3f40d826294845ba9ab3de678b22b4df, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681858519092 2023-04-18 22:55:28,232 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 56969893b1a14da78fad0c0e4660a5cc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681858521887 2023-04-18 22:55:28,233 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 94a0fb0487604e4b89e5af6ec310bddb, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681858521252 2023-04-18 22:55:28,234 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 5c7e1f3386f14fbaaa3228cea7922749, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1681858524039 2023-04-18 22:55:28,234 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 23aff79838cd4625878921e7b4f97315, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681858521887 2023-04-18 22:55:28,234 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 765f294c99dc47039a099faadff83f61, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1681858524680 2023-04-18 22:55:28,234 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting a9a9a5e595a44e169f82fec3b0d30950, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1681858524039 2023-04-18 22:55:28,234 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 261a63ee3bb74c2a9f1326f817e241a3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681858525832 2023-04-18 22:55:28,234 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting ac22920fea7c46b9adc4609bb575ac1e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1681858524680 2023-04-18 22:55:28,235 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 8beb23785ff442fca480387cf5483c2c, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681858525832 2023-04-18 22:55:28,245 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a8c3530d711f58ef72306d7c2b1ec7c#B#compaction#359 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:28,258 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1a8c3530d711f58ef72306d7c2b1ec7c#A#compaction#360 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:28,262 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/6a7cb38817d34108b9c62d29321a487b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/6a7cb38817d34108b9c62d29321a487b 2023-04-18 22:55:28,269 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 1a8c3530d711f58ef72306d7c2b1ec7c/B of 1a8c3530d711f58ef72306d7c2b1ec7c into 6a7cb38817d34108b9c62d29321a487b(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:28,269 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:28,269 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c., storeName=1a8c3530d711f58ef72306d7c2b1ec7c/B, priority=9, startTime=1681858528229; duration=0sec 2023-04-18 22:55:28,269 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:28,269 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-18 22:55:28,273 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84342 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-18 22:55:28,273 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 1a8c3530d711f58ef72306d7c2b1ec7c/C is initiating minor compaction (all files) 2023-04-18 22:55:28,273 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1a8c3530d711f58ef72306d7c2b1ec7c/C in TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:28,273 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/5537f333033544ae8a540670c0965287, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/4b17ec3b9a8d439e9bfd7a929a3b4a62, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/6b99e9b595d841be90e4f3d08a95423d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/2db4b2eda1164545aec6d4255657b8d4, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/f8415ae961bb44538fe65bbac798f356, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/0fe297fe7758487f9b136a587aabbf1d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/65bdfd86d5c244d6955622a0a6ea6643] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp, totalSize=82.4 K 2023-04-18 22:55:28,274 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 5537f333033544ae8a540670c0965287, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681858517914 2023-04-18 22:55:28,274 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 4b17ec3b9a8d439e9bfd7a929a3b4a62, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681858519092 2023-04-18 22:55:28,275 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 6b99e9b595d841be90e4f3d08a95423d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681858521252 2023-04-18 22:55:28,275 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 2db4b2eda1164545aec6d4255657b8d4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681858521887 2023-04-18 22:55:28,275 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting f8415ae961bb44538fe65bbac798f356, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1681858524039 2023-04-18 22:55:28,276 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 0fe297fe7758487f9b136a587aabbf1d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1681858524680 2023-04-18 22:55:28,277 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 65bdfd86d5c244d6955622a0a6ea6643, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681858525832 2023-04-18 22:55:28,292 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 1a8c3530d711f58ef72306d7c2b1ec7c#C#compaction#361 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:28,325 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:28,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:28,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:28,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:28,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:28,330 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:28,330 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:28,330 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:28,330 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:28,330 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,330 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:28,331 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:28,331 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:28,332 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:28,332 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:28,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:28,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:28,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:28,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:28,340 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:28,340 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:28,340 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:28,342 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:28,342 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:28,342 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:28,342 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:28,342 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a8c3530d711f58ef72306d7c2b1ec7c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 22:55:28,343 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=A 2023-04-18 22:55:28,343 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:28,343 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=B 2023-04-18 22:55:28,343 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:28,343 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=C 2023-04-18 22:55:28,343 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:28,710 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/4db2e98e1806479484811015f9051f76 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/4db2e98e1806479484811015f9051f76 2023-04-18 22:55:28,713 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 1a8c3530d711f58ef72306d7c2b1ec7c/C of 1a8c3530d711f58ef72306d7c2b1ec7c into 4db2e98e1806479484811015f9051f76(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:28,713 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:28,713 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c., storeName=1a8c3530d711f58ef72306d7c2b1ec7c/C, priority=9, startTime=1681858528229; duration=0sec 2023-04-18 22:55:28,713 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:28,766 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418a8b5e4620bd045bda382f58f8694ad44 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a8b5e4620bd045bda382f58f8694ad44 2023-04-18 22:55:29,103 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/cf70d77afd6f49768ff90f18384fd866 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/cf70d77afd6f49768ff90f18384fd866 2023-04-18 22:55:29,107 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 1a8c3530d711f58ef72306d7c2b1ec7c/A of 1a8c3530d711f58ef72306d7c2b1ec7c into cf70d77afd6f49768ff90f18384fd866(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:29,107 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:29,107 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c., storeName=1a8c3530d711f58ef72306d7c2b1ec7c/A, priority=9, startTime=1681858528228; duration=0sec 2023-04-18 22:55:29,107 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:29,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:29,148 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:29,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:29,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41260 deadline: 1681858589159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:29,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41194 deadline: 1681858589160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:29,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41256 deadline: 1681858589161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:29,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41264 deadline: 1681858589162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:29,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41244 deadline: 1681858589163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,173 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=237, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/2c2ffe8f69d147c891e8e6ff900eb032 2023-04-18 22:55:29,182 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/cad074baa08046628a873c7d41102860 2023-04-18 22:55:29,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:29,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41256 deadline: 1681858589268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:29,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41264 deadline: 1681858589268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:29,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41244 deadline: 1681858589272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:29,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41256 deadline: 1681858589473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:29,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41264 deadline: 1681858589473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:29,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41244 deadline: 1681858589479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,592 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/1570f1a91c9f405daa83ad400d7a22b9 2023-04-18 22:55:29,599 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/2c2ffe8f69d147c891e8e6ff900eb032 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/2c2ffe8f69d147c891e8e6ff900eb032 2023-04-18 22:55:29,604 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/2c2ffe8f69d147c891e8e6ff900eb032, entries=150, sequenceid=237, filesize=25.3 K 2023-04-18 22:55:29,605 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/cad074baa08046628a873c7d41102860 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/cad074baa08046628a873c7d41102860 2023-04-18 22:55:29,609 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/cad074baa08046628a873c7d41102860, entries=150, sequenceid=237, filesize=11.8 K 2023-04-18 22:55:29,610 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/1570f1a91c9f405daa83ad400d7a22b9 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/1570f1a91c9f405daa83ad400d7a22b9 2023-04-18 22:55:29,615 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/1570f1a91c9f405daa83ad400d7a22b9, entries=150, sequenceid=237, filesize=11.8 K 2023-04-18 22:55:29,617 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 1a8c3530d711f58ef72306d7c2b1ec7c in 1275ms, sequenceid=237, compaction requested=false 2023-04-18 22:55:29,617 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:29,617 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:29,617 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:29,617 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:29,617 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,618 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:29,618 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:29,620 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:29,620 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:29,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:29,622 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:29,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ab91988[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:29,622 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:29,623 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:29,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:29,624 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:29,624 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:29,624 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:29,629 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:29,629 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:29,629 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,629 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:29,629 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:29,629 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:29,629 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:29,629 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:29,629 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:29,629 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:29,629 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:29,631 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:29,631 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:29,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:29,631 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:29,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:29,632 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:29,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,637 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,637 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:29,637 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:29,637 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:29,637 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:29,637 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:29,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:29,637 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:29,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:29,637 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:29,637 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:29,637 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:29,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:29,637 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:29,637 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:29,737 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:29,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:29,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:29,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:29,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:29,742 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:29,742 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:29,742 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:29,742 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:29,742 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:29,743 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:29,746 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,746 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:29,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:29,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:29,749 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,749 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:29,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:29,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:29,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:29,750 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:29,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:29,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:29,751 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:29,751 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:29,751 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a8c3530d711f58ef72306d7c2b1ec7c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 22:55:29,751 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=A 2023-04-18 22:55:29,752 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:29,752 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=B 2023-04-18 22:55:29,752 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:29,752 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=C 2023-04-18 22:55:29,752 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:29,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:29,779 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:29,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:29,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41256 deadline: 1681858589818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:29,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41264 deadline: 1681858589823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:29,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41244 deadline: 1681858589824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:29,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41256 deadline: 1681858589927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:29,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41264 deadline: 1681858589934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:29,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:29,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41244 deadline: 1681858589934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:30,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:30,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41256 deadline: 1681858590131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:30,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:30,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41264 deadline: 1681858590138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:30,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:30,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41244 deadline: 1681858590139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:30,167 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418d903d4fcae384f7c9fc99233400bee3b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418d903d4fcae384f7c9fc99233400bee3b 2023-04-18 22:55:30,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:30,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41256 deadline: 1681858590440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:30,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:30,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41264 deadline: 1681858590443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:30,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:30,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41244 deadline: 1681858590445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:30,573 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=252, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/2d3b605d0b0b4bea9e7b1624987f4757 2023-04-18 22:55:30,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:30,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41256 deadline: 1681858590944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:30,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:30,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41264 deadline: 1681858590945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:30,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:30,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41244 deadline: 1681858590951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:30,982 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/801f273fb4d44409ba47209d8e52332d 2023-04-18 22:55:30,992 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/81d7687f644e454a84240c6aeb7ccd9e 2023-04-18 22:55:30,995 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/2d3b605d0b0b4bea9e7b1624987f4757 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/2d3b605d0b0b4bea9e7b1624987f4757 2023-04-18 22:55:30,999 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/2d3b605d0b0b4bea9e7b1624987f4757, entries=150, sequenceid=252, filesize=25.3 K 2023-04-18 22:55:31,000 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/801f273fb4d44409ba47209d8e52332d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/801f273fb4d44409ba47209d8e52332d 2023-04-18 22:55:31,003 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/801f273fb4d44409ba47209d8e52332d, entries=150, sequenceid=252, filesize=11.8 K 2023-04-18 22:55:31,004 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/81d7687f644e454a84240c6aeb7ccd9e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/81d7687f644e454a84240c6aeb7ccd9e 2023-04-18 22:55:31,008 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/81d7687f644e454a84240c6aeb7ccd9e, entries=150, sequenceid=252, filesize=11.8 K 2023-04-18 22:55:31,009 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1a8c3530d711f58ef72306d7c2b1ec7c in 1258ms, sequenceid=252, compaction requested=true 2023-04-18 22:55:31,009 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:31,009 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:31,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:31,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:31,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:31,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:31,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:31,012 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:31,012 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:31,012 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:31,012 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:31,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:31,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57aed5bb[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:31,014 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:31,014 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:31,016 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:31,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:31,017 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:31,017 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:31,018 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:31,018 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:31,018 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:31,018 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:31,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:31,021 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:31,021 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:31,021 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:31,021 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:31,021 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:31,023 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:31,023 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:31,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:31,024 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:31,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:31,024 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:31,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:31,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:31,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:31,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:31,027 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:31,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:31,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:31,030 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:31,030 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:31,030 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:31,030 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:31,030 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:31,030 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:31,030 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:31,030 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:31,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:31,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:31,030 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:31,031 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:31,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:31,031 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:31,031 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:31,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:31,131 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:31,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:31,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:31,133 DEBUG [Thread-2449] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e86d9 to 127.0.0.1:50825 2023-04-18 22:55:31,133 DEBUG [Thread-2449] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:31,134 DEBUG [Thread-2451] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x711ebaf2 to 127.0.0.1:50825 2023-04-18 22:55:31,134 DEBUG [Thread-2451] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:31,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:31,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:31,135 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:31,135 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:31,135 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:31,135 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:31,135 DEBUG [Thread-2447] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21128196 to 127.0.0.1:50825 2023-04-18 22:55:31,135 DEBUG [Thread-2447] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:31,137 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:31,137 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:31,139 DEBUG [Thread-2453] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a7503d4 to 127.0.0.1:50825 2023-04-18 22:55:31,139 DEBUG [Thread-2455] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7eae4191 to 127.0.0.1:50825 2023-04-18 22:55:31,139 DEBUG [Thread-2453] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:31,139 DEBUG [Thread-2455] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:31,142 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:31,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:31,142 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:31,142 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:31,142 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:31,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:31,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:31,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:31,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:31,143 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:31,143 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:31,143 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:31,143 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:31,143 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:31,143 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:31,143 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. started... 2023-04-18 22:55:31,143 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1a8c3530d711f58ef72306d7c2b1ec7c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 22:55:31,144 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=A 2023-04-18 22:55:31,144 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:31,144 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=B 2023-04-18 22:55:31,144 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:31,144 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=C 2023-04-18 22:55:31,144 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:31,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:31,171 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. as already flushing 2023-04-18 22:55:31,171 DEBUG [Thread-2444] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e01d5e5 to 127.0.0.1:50825 2023-04-18 22:55:31,171 DEBUG [Thread-2444] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:31,184 DEBUG [Thread-2436] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x445919fd to 127.0.0.1:50825 2023-04-18 22:55:31,184 DEBUG [Thread-2436] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:31,559 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304187f5e77bc60534a5fbdb8ff8238e88a13 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187f5e77bc60534a5fbdb8ff8238e88a13 2023-04-18 22:55:31,955 DEBUG [Thread-2440] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69100738 to 127.0.0.1:50825 2023-04-18 22:55:31,955 DEBUG [Thread-2440] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:31,958 DEBUG [Thread-2442] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f603c27 to 127.0.0.1:50825 2023-04-18 22:55:31,958 DEBUG [Thread-2442] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:31,963 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=271, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/f112a503540c4b1abcae9d4edf0ca12c 2023-04-18 22:55:31,964 DEBUG [Thread-2438] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d010713 to 127.0.0.1:50825 2023-04-18 22:55:31,964 DEBUG [Thread-2438] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:32,372 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/089b68addd244d3a819e84c29c5cb2d1 2023-04-18 22:55:32,780 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/b4c08ddfc0d34526a6a7ea96d5645dc4 2023-04-18 22:55:32,784 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/f112a503540c4b1abcae9d4edf0ca12c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/f112a503540c4b1abcae9d4edf0ca12c 2023-04-18 22:55:32,791 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/f112a503540c4b1abcae9d4edf0ca12c, entries=150, sequenceid=271, filesize=25.4 K 2023-04-18 22:55:32,795 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/089b68addd244d3a819e84c29c5cb2d1 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/089b68addd244d3a819e84c29c5cb2d1 2023-04-18 22:55:32,805 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/089b68addd244d3a819e84c29c5cb2d1, entries=150, sequenceid=271, filesize=11.9 K 2023-04-18 22:55:32,805 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/b4c08ddfc0d34526a6a7ea96d5645dc4 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/b4c08ddfc0d34526a6a7ea96d5645dc4 2023-04-18 22:55:32,810 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/b4c08ddfc0d34526a6a7ea96d5645dc4, entries=150, sequenceid=271, filesize=11.9 K 2023-04-18 22:55:32,811 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=33.54 KB/34350 for 1a8c3530d711f58ef72306d7c2b1ec7c in 1668ms, sequenceid=271, compaction requested=true 2023-04-18 22:55:32,811 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:32,811 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:32,811 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:32,811 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:32,811 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:32,811 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:32,811 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:32,813 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:32,813 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:32,814 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:32,814 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:32,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:32,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c5289d3[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:32,815 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:32,816 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:32,818 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:32,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:32,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:32,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:32,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:32,819 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:32,819 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:32,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:32,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:32,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:32,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:32,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:32,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:32,824 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:32,824 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:32,824 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:32,824 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:32,824 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:32,826 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:32,826 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:32,826 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:32,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:32,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:32,828 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:32,833 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:32,840 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:32,840 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:32,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:32,840 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:32,840 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:32,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:32,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:32,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:32,840 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:32,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:32,840 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:32,840 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:32,840 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:32,840 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:32,840 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:32,844 DEBUG [Thread-2446] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:32,844 DEBUG [Thread-2446] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:32,944 DEBUG [Thread-2446] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:32,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:32,945 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-18 22:55:32,945 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 40 2023-04-18 22:55:32,945 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 45 2023-04-18 22:55:32,945 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 50 2023-04-18 22:55:32,945 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 44 2023-04-18 22:55:32,945 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 39 2023-04-18 22:55:32,945 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-18 22:55:32,945 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-18 22:55:32,945 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(402): scanned 1983 2023-04-18 22:55:32,945 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(403): verified 5949 rows 2023-04-18 22:55:32,945 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(402): scanned 1970 2023-04-18 22:55:32,945 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(403): verified 5910 rows 2023-04-18 22:55:32,945 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(402): scanned 1967 2023-04-18 22:55:32,945 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(403): verified 5901 rows 2023-04-18 22:55:32,946 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(402): scanned 1972 2023-04-18 22:55:32,946 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(403): verified 5913 rows 2023-04-18 22:55:32,946 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(402): scanned 1984 2023-04-18 22:55:32,946 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(403): verified 5948 rows 2023-04-18 22:55:32,946 INFO [Listener at localhost/36651] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-18 22:55:32,946 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bb27fd4 to 127.0.0.1:50825 2023-04-18 22:55:32,946 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:32,947 INFO [Listener at localhost/36651] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-18 22:55:32,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-18 22:55:32,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:32,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-18 22:55:32,952 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858532952"}]},"ts":"1681858532952"} 2023-04-18 22:55:32,953 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-18 22:55:32,955 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-18 22:55:32,955 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a8c3530d711f58ef72306d7c2b1ec7c, UNASSIGN}] 2023-04-18 22:55:32,957 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a8c3530d711f58ef72306d7c2b1ec7c, UNASSIGN 2023-04-18 22:55:32,957 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=1a8c3530d711f58ef72306d7c2b1ec7c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:32,957 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858532957"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858532957"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858532957"}]},"ts":"1681858532957"} 2023-04-18 22:55:32,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:55:33,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-18 22:55:33,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:33,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1a8c3530d711f58ef72306d7c2b1ec7c, disabling compactions & flushes 2023-04-18 22:55:33,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:33,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:33,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. after waiting 0 ms 2023-04-18 22:55:33,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:33,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1a8c3530d711f58ef72306d7c2b1ec7c 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-18 22:55:33,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=A 2023-04-18 22:55:33,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:33,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=B 2023-04-18 22:55:33,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:33,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1a8c3530d711f58ef72306d7c2b1ec7c, store=C 2023-04-18 22:55:33,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:33,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-18 22:55:33,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304185ed0bc10f6524670a320d6b351af8b55 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304185ed0bc10f6524670a320d6b351af8b55 2023-04-18 22:55:33,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=279, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/57e330e84e1d4c6dace48e30110cb080 2023-04-18 22:55:33,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-18 22:55:33,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/2ac445fe072c412e82f39c794eb5a8dc 2023-04-18 22:55:34,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-18 22:55:34,354 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/122891526c0c443da490db325c817fb3 2023-04-18 22:55:34,357 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/A/57e330e84e1d4c6dace48e30110cb080 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/57e330e84e1d4c6dace48e30110cb080 2023-04-18 22:55:34,360 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/57e330e84e1d4c6dace48e30110cb080, entries=150, sequenceid=279, filesize=25.4 K 2023-04-18 22:55:34,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/B/2ac445fe072c412e82f39c794eb5a8dc as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/2ac445fe072c412e82f39c794eb5a8dc 2023-04-18 22:55:34,365 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/2ac445fe072c412e82f39c794eb5a8dc, entries=150, sequenceid=279, filesize=11.9 K 2023-04-18 22:55:34,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/.tmp/C/122891526c0c443da490db325c817fb3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/122891526c0c443da490db325c817fb3 2023-04-18 22:55:34,369 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/122891526c0c443da490db325c817fb3, entries=150, sequenceid=279, filesize=11.9 K 2023-04-18 22:55:34,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 1a8c3530d711f58ef72306d7c2b1ec7c in 1260ms, sequenceid=279, compaction requested=true 2023-04-18 22:55:34,371 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/989a18cff2ce4570b0b3cad90946b718, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/95b831c3145a45b69e4ea608cc507227, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/03424cafea584e71ab3a50e220c8887d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/e0278fa4ee18486398674b96a95cb9df, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/1769f5ebfe084157938ddf411de17632, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/42008a786af74cc6a51a8231afd33a81, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/69ca83b9391c4097bb7d99d73b211926, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/4d59b3d64bdc47bea5aa7cb2184de198, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/3f40d826294845ba9ab3de678b22b4df, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/94a0fb0487604e4b89e5af6ec310bddb, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/23aff79838cd4625878921e7b4f97315, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/a9a9a5e595a44e169f82fec3b0d30950, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/ac22920fea7c46b9adc4609bb575ac1e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/8beb23785ff442fca480387cf5483c2c] to archive 2023-04-18 22:55:34,371 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:55:34,373 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/989a18cff2ce4570b0b3cad90946b718 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/989a18cff2ce4570b0b3cad90946b718 2023-04-18 22:55:34,374 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/95b831c3145a45b69e4ea608cc507227 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/95b831c3145a45b69e4ea608cc507227 2023-04-18 22:55:34,375 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/03424cafea584e71ab3a50e220c8887d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/03424cafea584e71ab3a50e220c8887d 2023-04-18 22:55:34,377 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/e0278fa4ee18486398674b96a95cb9df to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/e0278fa4ee18486398674b96a95cb9df 2023-04-18 22:55:34,378 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/1769f5ebfe084157938ddf411de17632 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/1769f5ebfe084157938ddf411de17632 2023-04-18 22:55:34,379 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/42008a786af74cc6a51a8231afd33a81 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/42008a786af74cc6a51a8231afd33a81 2023-04-18 22:55:34,380 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/69ca83b9391c4097bb7d99d73b211926 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/69ca83b9391c4097bb7d99d73b211926 2023-04-18 22:55:34,381 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/4d59b3d64bdc47bea5aa7cb2184de198 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/4d59b3d64bdc47bea5aa7cb2184de198 2023-04-18 22:55:34,382 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/3f40d826294845ba9ab3de678b22b4df to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/3f40d826294845ba9ab3de678b22b4df 2023-04-18 22:55:34,383 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/94a0fb0487604e4b89e5af6ec310bddb to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/94a0fb0487604e4b89e5af6ec310bddb 2023-04-18 22:55:34,384 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/23aff79838cd4625878921e7b4f97315 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/23aff79838cd4625878921e7b4f97315 2023-04-18 22:55:34,385 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/a9a9a5e595a44e169f82fec3b0d30950 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/a9a9a5e595a44e169f82fec3b0d30950 2023-04-18 22:55:34,386 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/ac22920fea7c46b9adc4609bb575ac1e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/ac22920fea7c46b9adc4609bb575ac1e 2023-04-18 22:55:34,387 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/8beb23785ff442fca480387cf5483c2c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/8beb23785ff442fca480387cf5483c2c 2023-04-18 22:55:34,389 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/55311bdfc65b49cda69823225840a342, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/1339686c5fe749c1bca3a3c4ad76f89a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/4e3a5d36b73549bd98e1b56c3fac56f9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5185f412159943f0b08531fd97409490, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/ebd79e873d4642e2bbebb23b69dadf9a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/65ee99ddf4e8487c826f14099aca1b79, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/546a77eb26004ce7bb4bd9fb8eea44bb, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/ae997fa971214d769d5ed920094d0f32, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5468d372759d420f914b0d03ee875923, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/49b956d9a915423b9ad8549b2663567b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/56969893b1a14da78fad0c0e4660a5cc, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5c7e1f3386f14fbaaa3228cea7922749, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/765f294c99dc47039a099faadff83f61, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/261a63ee3bb74c2a9f1326f817e241a3] to archive 2023-04-18 22:55:34,389 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:55:34,391 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/55311bdfc65b49cda69823225840a342 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/55311bdfc65b49cda69823225840a342 2023-04-18 22:55:34,392 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/1339686c5fe749c1bca3a3c4ad76f89a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/1339686c5fe749c1bca3a3c4ad76f89a 2023-04-18 22:55:34,393 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/4e3a5d36b73549bd98e1b56c3fac56f9 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/4e3a5d36b73549bd98e1b56c3fac56f9 2023-04-18 22:55:34,394 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5185f412159943f0b08531fd97409490 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5185f412159943f0b08531fd97409490 2023-04-18 22:55:34,395 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/ebd79e873d4642e2bbebb23b69dadf9a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/ebd79e873d4642e2bbebb23b69dadf9a 2023-04-18 22:55:34,396 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/65ee99ddf4e8487c826f14099aca1b79 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/65ee99ddf4e8487c826f14099aca1b79 2023-04-18 22:55:34,397 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/546a77eb26004ce7bb4bd9fb8eea44bb to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/546a77eb26004ce7bb4bd9fb8eea44bb 2023-04-18 22:55:34,399 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/ae997fa971214d769d5ed920094d0f32 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/ae997fa971214d769d5ed920094d0f32 2023-04-18 22:55:34,400 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5468d372759d420f914b0d03ee875923 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5468d372759d420f914b0d03ee875923 2023-04-18 22:55:34,401 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/49b956d9a915423b9ad8549b2663567b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/49b956d9a915423b9ad8549b2663567b 2023-04-18 22:55:34,402 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/56969893b1a14da78fad0c0e4660a5cc to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/56969893b1a14da78fad0c0e4660a5cc 2023-04-18 22:55:34,403 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5c7e1f3386f14fbaaa3228cea7922749 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/5c7e1f3386f14fbaaa3228cea7922749 2023-04-18 22:55:34,404 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/765f294c99dc47039a099faadff83f61 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/765f294c99dc47039a099faadff83f61 2023-04-18 22:55:34,404 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/261a63ee3bb74c2a9f1326f817e241a3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/261a63ee3bb74c2a9f1326f817e241a3 2023-04-18 22:55:34,406 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/e5ed8b8a8d194287a6c64e3159ae6c37, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/fc606345420d4f15ae74f14caf143a09, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/6f8a9b99158b4d08b2f0489acfd4f8c3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/f2203046660249ac938441cf9d55f965, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/167ceab6b35a48f9a0984c0f6c9b91f6, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/d533e7066f8840e99f19c13ebfcc0791, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/5537f333033544ae8a540670c0965287, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/55defdabcabc46f9be0bd4fc71d75a1b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/4b17ec3b9a8d439e9bfd7a929a3b4a62, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/6b99e9b595d841be90e4f3d08a95423d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/2db4b2eda1164545aec6d4255657b8d4, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/f8415ae961bb44538fe65bbac798f356, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/0fe297fe7758487f9b136a587aabbf1d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/65bdfd86d5c244d6955622a0a6ea6643] to archive 2023-04-18 22:55:34,407 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:55:34,408 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/e5ed8b8a8d194287a6c64e3159ae6c37 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/e5ed8b8a8d194287a6c64e3159ae6c37 2023-04-18 22:55:34,409 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/fc606345420d4f15ae74f14caf143a09 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/fc606345420d4f15ae74f14caf143a09 2023-04-18 22:55:34,410 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/6f8a9b99158b4d08b2f0489acfd4f8c3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/6f8a9b99158b4d08b2f0489acfd4f8c3 2023-04-18 22:55:34,411 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/f2203046660249ac938441cf9d55f965 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/f2203046660249ac938441cf9d55f965 2023-04-18 22:55:34,411 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/167ceab6b35a48f9a0984c0f6c9b91f6 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/167ceab6b35a48f9a0984c0f6c9b91f6 2023-04-18 22:55:34,412 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/d533e7066f8840e99f19c13ebfcc0791 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/d533e7066f8840e99f19c13ebfcc0791 2023-04-18 22:55:34,413 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/5537f333033544ae8a540670c0965287 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/5537f333033544ae8a540670c0965287 2023-04-18 22:55:34,414 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/55defdabcabc46f9be0bd4fc71d75a1b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/55defdabcabc46f9be0bd4fc71d75a1b 2023-04-18 22:55:34,415 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/4b17ec3b9a8d439e9bfd7a929a3b4a62 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/4b17ec3b9a8d439e9bfd7a929a3b4a62 2023-04-18 22:55:34,415 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/6b99e9b595d841be90e4f3d08a95423d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/6b99e9b595d841be90e4f3d08a95423d 2023-04-18 22:55:34,416 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/2db4b2eda1164545aec6d4255657b8d4 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/2db4b2eda1164545aec6d4255657b8d4 2023-04-18 22:55:34,417 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/f8415ae961bb44538fe65bbac798f356 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/f8415ae961bb44538fe65bbac798f356 2023-04-18 22:55:34,418 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/0fe297fe7758487f9b136a587aabbf1d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/0fe297fe7758487f9b136a587aabbf1d 2023-04-18 22:55:34,419 DEBUG [StoreCloser-TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/65bdfd86d5c244d6955622a0a6ea6643 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/65bdfd86d5c244d6955622a0a6ea6643 2023-04-18 22:55:34,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/recovered.edits/282.seqid, newMaxSeqId=282, maxSeqId=4 2023-04-18 22:55:34,424 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c. 2023-04-18 22:55:34,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1a8c3530d711f58ef72306d7c2b1ec7c: 2023-04-18 22:55:34,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:34,425 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=1a8c3530d711f58ef72306d7c2b1ec7c, regionState=CLOSED 2023-04-18 22:55:34,426 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858534425"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681858534425"}]},"ts":"1681858534425"} 2023-04-18 22:55:34,427 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-18 22:55:34,428 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 1a8c3530d711f58ef72306d7c2b1ec7c, server=jenkins-hbase4.apache.org,35971,1681858424869 in 1.4680 sec 2023-04-18 22:55:34,429 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-18 22:55:34,429 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1a8c3530d711f58ef72306d7c2b1ec7c, UNASSIGN in 1.4730 sec 2023-04-18 22:55:34,429 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858534429"}]},"ts":"1681858534429"} 2023-04-18 22:55:34,430 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-18 22:55:34,433 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-18 22:55:34,434 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.4850 sec 2023-04-18 22:55:35,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-18 22:55:35,057 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-18 22:55:35,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-18 22:55:35,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:35,059 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:35,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-18 22:55:35,060 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:35,065 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:35,067 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/recovered.edits] 2023-04-18 22:55:35,069 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/2c2ffe8f69d147c891e8e6ff900eb032 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/2c2ffe8f69d147c891e8e6ff900eb032 2023-04-18 22:55:35,071 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/2d3b605d0b0b4bea9e7b1624987f4757 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/2d3b605d0b0b4bea9e7b1624987f4757 2023-04-18 22:55:35,072 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/57e330e84e1d4c6dace48e30110cb080 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/57e330e84e1d4c6dace48e30110cb080 2023-04-18 22:55:35,073 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/cf70d77afd6f49768ff90f18384fd866 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/cf70d77afd6f49768ff90f18384fd866 2023-04-18 22:55:35,074 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/f112a503540c4b1abcae9d4edf0ca12c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/A/f112a503540c4b1abcae9d4edf0ca12c 2023-04-18 22:55:35,076 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/089b68addd244d3a819e84c29c5cb2d1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/089b68addd244d3a819e84c29c5cb2d1 2023-04-18 22:55:35,077 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/2ac445fe072c412e82f39c794eb5a8dc to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/2ac445fe072c412e82f39c794eb5a8dc 2023-04-18 22:55:35,078 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/6a7cb38817d34108b9c62d29321a487b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/6a7cb38817d34108b9c62d29321a487b 2023-04-18 22:55:35,079 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/801f273fb4d44409ba47209d8e52332d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/801f273fb4d44409ba47209d8e52332d 2023-04-18 22:55:35,080 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/cad074baa08046628a873c7d41102860 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/B/cad074baa08046628a873c7d41102860 2023-04-18 22:55:35,082 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/122891526c0c443da490db325c817fb3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/122891526c0c443da490db325c817fb3 2023-04-18 22:55:35,082 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/1570f1a91c9f405daa83ad400d7a22b9 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/1570f1a91c9f405daa83ad400d7a22b9 2023-04-18 22:55:35,083 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/4db2e98e1806479484811015f9051f76 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/4db2e98e1806479484811015f9051f76 2023-04-18 22:55:35,084 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/81d7687f644e454a84240c6aeb7ccd9e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/81d7687f644e454a84240c6aeb7ccd9e 2023-04-18 22:55:35,085 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/b4c08ddfc0d34526a6a7ea96d5645dc4 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/C/b4c08ddfc0d34526a6a7ea96d5645dc4 2023-04-18 22:55:35,087 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/recovered.edits/282.seqid to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c/recovered.edits/282.seqid 2023-04-18 22:55:35,088 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/1a8c3530d711f58ef72306d7c2b1ec7c 2023-04-18 22:55:35,088 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 22:55:35,088 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-18 22:55:35,089 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-18 22:55:35,091 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041800533a8e0e284903af9834116b73310d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041800533a8e0e284903af9834116b73310d 2023-04-18 22:55:35,091 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304180d9111c25c734c238a86e5ecce6d905e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304180d9111c25c734c238a86e5ecce6d905e 2023-04-18 22:55:35,092 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041810dd632060cd461a9ccebcdae35efbbe to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041810dd632060cd461a9ccebcdae35efbbe 2023-04-18 22:55:35,093 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418194d3e006c2f47e082203b30e7f0b78a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418194d3e006c2f47e082203b30e7f0b78a 2023-04-18 22:55:35,094 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304185ed0bc10f6524670a320d6b351af8b55 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304185ed0bc10f6524670a320d6b351af8b55 2023-04-18 22:55:35,094 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418653268c60faa4c9f8ef3efe9026c87ef to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418653268c60faa4c9f8ef3efe9026c87ef 2023-04-18 22:55:35,095 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186c9b5e4de5344e9287ec442a023fe85b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186c9b5e4de5344e9287ec442a023fe85b 2023-04-18 22:55:35,096 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187f5e77bc60534a5fbdb8ff8238e88a13 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187f5e77bc60534a5fbdb8ff8238e88a13 2023-04-18 22:55:35,097 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a8b5e4620bd045bda382f58f8694ad44 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418a8b5e4620bd045bda382f58f8694ad44 2023-04-18 22:55:35,098 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418b76aaf8d05a5483682f275164bd61b7a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418b76aaf8d05a5483682f275164bd61b7a 2023-04-18 22:55:35,099 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418bba62ceff7334e9cb443d55b39508644 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418bba62ceff7334e9cb443d55b39508644 2023-04-18 22:55:35,099 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418bcb24edc328d4ae5ad6ade0d12d0e966 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418bcb24edc328d4ae5ad6ade0d12d0e966 2023-04-18 22:55:35,100 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418d903d4fcae384f7c9fc99233400bee3b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418d903d4fcae384f7c9fc99233400bee3b 2023-04-18 22:55:35,101 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e066f43cde524156a793516770fb0885 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e066f43cde524156a793516770fb0885 2023-04-18 22:55:35,102 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ee4a5bbd954f4f2ba4575b9d03460f25 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418ee4a5bbd954f4f2ba4575b9d03460f25 2023-04-18 22:55:35,102 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f0a122e53e5e4a5ba718c94f20a27234 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f0a122e53e5e4a5ba718c94f20a27234 2023-04-18 22:55:35,103 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f2798b13ae934d068fbcde573f3f4e4c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f2798b13ae934d068fbcde573f3f4e4c 2023-04-18 22:55:35,103 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-18 22:55:35,105 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:35,106 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-18 22:55:35,108 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-18 22:55:35,108 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:35,108 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-18 22:55:35,109 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681858535108"}]},"ts":"9223372036854775807"} 2023-04-18 22:55:35,110 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-18 22:55:35,110 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1a8c3530d711f58ef72306d7c2b1ec7c, NAME => 'TestAcidGuarantees,,1681858508058.1a8c3530d711f58ef72306d7c2b1ec7c.', STARTKEY => '', ENDKEY => ''}] 2023-04-18 22:55:35,110 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-18 22:55:35,110 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681858535110"}]},"ts":"9223372036854775807"} 2023-04-18 22:55:35,111 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-18 22:55:35,120 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:35,121 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 63 msec 2023-04-18 22:55:35,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-18 22:55:35,161 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-18 22:55:35,175 INFO [Listener at localhost/36651] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 229), OpenFileDescriptor=521 (was 566), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 399), ProcessCount=169 (was 169), AvailableMemoryMB=6201 (was 6254) 2023-04-18 22:55:35,191 INFO [Listener at localhost/36651] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=521, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=169, AvailableMemoryMB=6201 2023-04-18 22:55:35,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 22:55:35,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 22:55:35,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:35,196 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-18 22:55:35,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-18 22:55:35,197 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-18 22:55:35,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-18 22:55:35,203 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:35,204 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0 empty. 2023-04-18 22:55:35,204 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:35,204 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 22:55:35,217 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 22:55:35,219 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 71e1d4dea1ebcdcdd7da8f1e7108edb0, NAME => 'TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp 2023-04-18 22:55:35,231 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:55:35,231 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 71e1d4dea1ebcdcdd7da8f1e7108edb0, disabling compactions & flushes 2023-04-18 22:55:35,231 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:35,231 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:35,231 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. after waiting 0 ms 2023-04-18 22:55:35,231 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:35,231 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:35,231 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:35,235 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-18 22:55:35,236 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858535236"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681858535236"}]},"ts":"1681858535236"} 2023-04-18 22:55:35,239 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-18 22:55:35,240 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-18 22:55:35,240 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858535240"}]},"ts":"1681858535240"} 2023-04-18 22:55:35,241 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-18 22:55:35,244 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=71e1d4dea1ebcdcdd7da8f1e7108edb0, ASSIGN}] 2023-04-18 22:55:35,247 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=71e1d4dea1ebcdcdd7da8f1e7108edb0, ASSIGN 2023-04-18 22:55:35,247 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=71e1d4dea1ebcdcdd7da8f1e7108edb0, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35971,1681858424869; forceNewPlan=false, retain=false 2023-04-18 22:55:35,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-18 22:55:35,399 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=71e1d4dea1ebcdcdd7da8f1e7108edb0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,399 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858535399"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858535399"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858535399"}]},"ts":"1681858535399"} 2023-04-18 22:55:35,400 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:55:35,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-18 22:55:35,555 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:35,555 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 71e1d4dea1ebcdcdd7da8f1e7108edb0, NAME => 'TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.', STARTKEY => '', ENDKEY => ''} 2023-04-18 22:55:35,555 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:35,555 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:55:35,555 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:35,555 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:35,556 INFO [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:35,557 DEBUG [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A 2023-04-18 22:55:35,557 DEBUG [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A 2023-04-18 22:55:35,558 INFO [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:55:35,558 INFO [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 71e1d4dea1ebcdcdd7da8f1e7108edb0 columnFamilyName A 2023-04-18 22:55:35,558 INFO [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] regionserver.HStore(310): Store=71e1d4dea1ebcdcdd7da8f1e7108edb0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:55:35,558 INFO [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:35,559 DEBUG [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B 2023-04-18 22:55:35,559 DEBUG [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B 2023-04-18 22:55:35,560 INFO [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:55:35,560 INFO [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 71e1d4dea1ebcdcdd7da8f1e7108edb0 columnFamilyName B 2023-04-18 22:55:35,560 INFO [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] regionserver.HStore(310): Store=71e1d4dea1ebcdcdd7da8f1e7108edb0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:55:35,560 INFO [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:35,561 DEBUG [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C 2023-04-18 22:55:35,561 DEBUG [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C 2023-04-18 22:55:35,562 INFO [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:55:35,562 INFO [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 71e1d4dea1ebcdcdd7da8f1e7108edb0 columnFamilyName C 2023-04-18 22:55:35,562 INFO [StoreOpener-71e1d4dea1ebcdcdd7da8f1e7108edb0-1] regionserver.HStore(310): Store=71e1d4dea1ebcdcdd7da8f1e7108edb0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:55:35,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:35,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:35,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:35,565 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 22:55:35,566 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:35,567 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 22:55:35,568 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 71e1d4dea1ebcdcdd7da8f1e7108edb0; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65916129, jitterRate=-0.017773136496543884}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 22:55:35,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:35,568 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., pid=49, masterSystemTime=1681858535551 2023-04-18 22:55:35,569 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:35,570 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:35,570 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=71e1d4dea1ebcdcdd7da8f1e7108edb0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,570 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858535570"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681858535570"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681858535570"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681858535570"}]},"ts":"1681858535570"} 2023-04-18 22:55:35,572 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-18 22:55:35,572 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 in 171 msec 2023-04-18 22:55:35,573 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-18 22:55:35,573 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=71e1d4dea1ebcdcdd7da8f1e7108edb0, ASSIGN in 328 msec 2023-04-18 22:55:35,574 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-18 22:55:35,574 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858535574"}]},"ts":"1681858535574"} 2023-04-18 22:55:35,575 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-18 22:55:35,577 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-18 22:55:35,578 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 383 msec 2023-04-18 22:55:35,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-18 22:55:35,800 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-18 22:55:35,801 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x37705401 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:35,807 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ec943a9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:35,808 DEBUG [Listener at localhost/36651] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:35,810 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48852, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:35,811 DEBUG [Listener at localhost/36651] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-18 22:55:35,813 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38616, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-18 22:55:35,814 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x15880ffb to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:35,819 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20ed9117, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:35,819 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x052203cb to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:35,823 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50002343, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:35,823 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x420f945a to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:35,827 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5726fb8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:35,828 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x51b8df2a to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:35,832 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f2b3c03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:35,833 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x67346c47 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:35,836 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e2be04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:35,837 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x067ea274 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:35,840 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63093e21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:35,841 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x78592df0 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:35,845 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b475d8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:35,845 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x659c68ab to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:35,850 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@549bef45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:35,850 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x680e773e to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:35,854 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ed183a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:35,855 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x0b8f652e to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:55:35,859 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10a344e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:55:35,864 DEBUG [hconnection-0x1e5dc817-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:35,864 DEBUG [hconnection-0x3f701e00-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:35,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:35,864 DEBUG [hconnection-0x1310fd23-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:35,865 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48868, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:35,871 DEBUG [hconnection-0x6db60bd1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:35,871 DEBUG [hconnection-0xe8f310d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:35,871 DEBUG [hconnection-0x21e25c6a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:35,872 DEBUG [hconnection-0x1ea26856-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:35,875 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48892, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:35,878 DEBUG [hconnection-0x51effd27-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:35,879 DEBUG [hconnection-0x36b53294-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:35,878 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48880, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:35,879 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48866, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:35,878 DEBUG [hconnection-0x3a6e5c01-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:55:35,879 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48890, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:35,880 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48928, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:35,880 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48912, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:35,883 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48956, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:35,884 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48934, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:35,889 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48950, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:55:35,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:35,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:35,908 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:35,908 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:35,908 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:35,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:35,908 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:35,908 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 22:55:35,909 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:35,909 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:35,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858535901, max timestamp=1681858535908Num uniques 859; 2023-04-18 22:55:35,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:55:35,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858535901, max timestamp=1681858535904Num uniques 859; 2023-04-18 22:55:35,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:55:35,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858535901, max timestamp=1681858535904Num uniques 859; 2023-04-18 22:55:35,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:55:35,911 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:35,911 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:35,911 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:35,911 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:35,911 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:35,911 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:35,911 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:35,911 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:35,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:35,912 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,912 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:35,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:35,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:35,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:35,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:35,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:35,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:35,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:35,915 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:35,915 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:35,915 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:35,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:35,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:35,916 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:35,916 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:35,916 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:35,916 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:35,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:35,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:35,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:35,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:35,923 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,923 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:35,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:35,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:35,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:35,924 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:35,924 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:35,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:35,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:35,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:35,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:35,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40ff8151[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:35,927 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:35,927 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:35,932 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:35,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:35,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:35,932 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,932 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:35,933 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:35,933 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:35,933 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:35,935 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:35,935 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,935 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:35,935 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:35,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:35,938 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:35,938 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:35,938 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:35,938 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:35,938 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:35,940 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:35,940 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:35,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:35,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:35,940 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:35,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:35,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:35,942 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681858535916, max timestamp=1681858535940Num uniques 859; 2023-04-18 22:55:35,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-18 22:55:35,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681858535916, max timestamp=1681858535940Num uniques 859; 2023-04-18 22:55:35,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681858535916, max timestamp=1681858535940Num uniques 859; 2023-04-18 22:55:35,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-18 22:55:35,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-18 22:55:35,944 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:35,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:35,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:35,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:35,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:35,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:35,947 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:35,947 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:35,947 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,947 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:35,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:35,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:35,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:35,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:35,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:35,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:35,947 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:35,947 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:35,947 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,947 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:35,947 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:35,947 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:35,947 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:35,947 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:35,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:35,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:35,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:35,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:35,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858595952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:35,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:35,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858595952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48868 deadline: 1681858595952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:35,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858595953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:35,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:35,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48912 deadline: 1681858595953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,048 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:36,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:36,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:36,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:36,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858596054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:36,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48868 deadline: 1681858596054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:36,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858596054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:36,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858596055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:36,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48912 deadline: 1681858596055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:36,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:36,059 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:36,059 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:36,059 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:36,059 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:36,060 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,060 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,061 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:36,061 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:36,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,062 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,062 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:36,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:36,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:36,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:36,069 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:36,069 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:36,069 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:36,071 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:36,071 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:36,071 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:36,071 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:36,073 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:36,073 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:36,073 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:36,073 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:36,073 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,073 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:36,073 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:36,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,075 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,076 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,076 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:36,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:36,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5298b59f[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:36,080 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:36,080 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,081 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,082 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:36,082 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:36,082 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:36,082 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,082 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:36,084 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:36,084 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:36,084 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:36,084 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:36,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:36,087 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:36,087 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:36,087 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:36,087 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:36,088 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:36,089 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,089 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,089 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:36,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:36,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:36,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,091 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,097 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,101 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,101 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,101 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,101 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,101 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:36,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:36,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:36,101 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,101 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:36,101 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,102 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,102 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:36,102 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:36,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:36,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:36,101 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:36,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:36,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:36,202 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:36,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:36,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:36,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:36,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:36,205 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:36,205 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:36,205 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:36,205 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:36,205 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,205 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,206 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,206 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:36,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:36,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:36,207 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,207 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:36,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:36,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:36,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:36,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:36,208 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:36,208 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:36,208 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:36,212 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:36,212 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:36,212 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:36,212 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,214 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:36,215 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:36,221 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,221 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,222 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,222 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:36,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73baa17a[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:36,225 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:36,225 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,226 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,226 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,226 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:36,226 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:36,226 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:36,226 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:36,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:36,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:36,231 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:36,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:36,233 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:36,233 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:36,233 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:36,233 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:36,233 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:36,236 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,236 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,236 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:36,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:36,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:36,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,237 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,238 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,240 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,240 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,240 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,240 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:36,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:36,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:36,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:36,241 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:36,240 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:36,240 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,241 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,241 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:36,241 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:36,241 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:36,241 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:36,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858596256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:36,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858596256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:36,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48868 deadline: 1681858596256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:36,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858596256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:36,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48912 deadline: 1681858596257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,341 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:36,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:36,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:36,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:36,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:36,344 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:36,345 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:36,345 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:36,345 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:36,345 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,345 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,346 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:36,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:36,347 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,347 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:36,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:36,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:36,348 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:36,348 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:36,348 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:36,348 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:36,348 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:36,348 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:36,348 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:36,348 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:36,348 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:36,348 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:36,348 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:36,348 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,348 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:36,348 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:36,350 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,350 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,350 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,350 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,351 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=24 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/cac5eb5d37794b9fa5e65ad1466169b8 2023-04-18 22:55:36,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:36,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b9eb2cf[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:36,352 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:36,352 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,354 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,354 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:36,355 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:36,355 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:36,355 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,355 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:36,356 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:36,356 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:36,356 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,356 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:36,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:36,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:36,361 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:36,361 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:36,361 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:36,361 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:36,361 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:36,362 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,362 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,362 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:36,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:36,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,364 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=24 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/ddf10b00c2854e6d99a7ac61f18737fc 2023-04-18 22:55:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,365 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,366 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:36,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:36,369 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:36,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:36,370 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:36,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:36,370 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:36,370 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:36,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:36,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:36,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:36,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:36,470 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:36,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:36,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:36,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:36,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:36,473 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:36,473 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:36,473 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:36,473 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:36,473 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,474 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,476 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,476 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:36,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:36,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:36,476 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,476 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:36,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:36,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:36,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:36,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:36,478 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:36,478 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:36,478 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:36,478 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:36,478 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:36,480 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,480 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,480 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,480 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:36,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@775757e4[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:36,482 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:36,482 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,483 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,483 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:36,483 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:36,483 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,483 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:36,483 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:36,484 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:36,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:36,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:36,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:36,492 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:36,492 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:36,492 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:36,492 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:36,492 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:36,499 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,499 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,500 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:36,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:36,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:36,500 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,501 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,504 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:36,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:36,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:36,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:36,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:36,504 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:36,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:36,505 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,505 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,505 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:36,505 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:36,505 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:36,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:36,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:36,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:36,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858596559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:36,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858596559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:36,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48868 deadline: 1681858596560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:36,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858596561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:36,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48912 deadline: 1681858596564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,605 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:36,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:36,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:36,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:36,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:36,609 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:36,609 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:36,609 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:36,609 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:36,609 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,609 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,611 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:36,611 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:36,611 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,611 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:36,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:36,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:36,612 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:36,612 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:36,612 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:36,612 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:36,612 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:36,612 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:36,613 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:36,613 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:36,613 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:36,613 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:36,613 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:36,613 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,613 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:36,613 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:36,614 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,615 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,615 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,615 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:36,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:36,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@607430dc[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:36,616 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:36,616 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,623 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,624 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:36,624 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:36,624 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:36,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:36,626 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:36,626 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:36,626 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,626 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:36,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:36,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:36,628 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:36,628 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:36,628 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:36,628 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:36,629 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:36,632 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,632 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,632 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:36,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,637 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:36,637 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:36,637 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:36,637 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:36,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:36,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:36,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:36,637 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,638 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:36,638 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,638 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:36,638 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:36,638 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,638 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,638 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,738 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:36,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:36,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:36,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:36,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:36,745 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:36,745 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:36,745 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:36,745 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:36,745 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,745 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,746 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:36,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:36,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:36,750 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,750 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,750 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:36,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:36,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:36,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:36,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:36,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:36,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:36,751 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:36,751 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:36,751 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:36,751 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:36,751 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:36,752 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:36,752 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:36,752 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,752 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:36,752 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:36,753 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,753 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,754 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,754 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:36,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46e3ecd9[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:36,755 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:36,755 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,756 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,757 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:36,757 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:36,757 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:36,757 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,757 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:36,764 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,764 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:36,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,764 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:36,765 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:36,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:36,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:36,767 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:36,767 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:36,767 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:36,767 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:36,767 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:36,769 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,769 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,769 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:36,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:36,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:36,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:36,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:36,769 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:36,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:36,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,771 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:36,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:36,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,774 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,774 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,774 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,774 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,774 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,774 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:36,774 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:36,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:36,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:36,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:36,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:36,774 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:36,774 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:36,774 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:36,775 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:36,775 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:36,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:36,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:36,776 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=24 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/02c94e12fdcf4d418c13ef2e2736aaa3 2023-04-18 22:55:36,779 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/cac5eb5d37794b9fa5e65ad1466169b8 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/cac5eb5d37794b9fa5e65ad1466169b8 2023-04-18 22:55:36,782 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/cac5eb5d37794b9fa5e65ad1466169b8, entries=150, sequenceid=24, filesize=11.6 K 2023-04-18 22:55:36,783 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/ddf10b00c2854e6d99a7ac61f18737fc as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ddf10b00c2854e6d99a7ac61f18737fc 2023-04-18 22:55:36,786 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ddf10b00c2854e6d99a7ac61f18737fc, entries=150, sequenceid=24, filesize=11.6 K 2023-04-18 22:55:36,787 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/02c94e12fdcf4d418c13ef2e2736aaa3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/02c94e12fdcf4d418c13ef2e2736aaa3 2023-04-18 22:55:36,789 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/02c94e12fdcf4d418c13ef2e2736aaa3, entries=150, sequenceid=24, filesize=11.6 K 2023-04-18 22:55:36,790 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~106.03 KB/108576, currentSize=161.02 KB/164880 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 882ms, sequenceid=24, compaction requested=false 2023-04-18 22:55:36,790 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-18 22:55:36,791 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:36,875 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:36,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:36,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:36,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:36,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:36,879 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:36,879 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:36,879 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:36,879 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:36,880 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,880 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:36,881 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:36,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,882 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:36,882 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:36,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:36,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:36,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:36,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:36,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:36,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:36,883 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:36,883 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:36,883 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:36,883 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:36,883 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:36,883 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:36,883 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:36,883 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=161.02 KB heapSize=441.23 KB 2023-04-18 22:55:36,884 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:36,884 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:36,884 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:36,884 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:36,884 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:36,884 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:36,904 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/9294f9e88d194d5eb56f57b6312dc48d 2023-04-18 22:55:37,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:37,068 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:37,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:37,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48868 deadline: 1681858597069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:37,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858597070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:37,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858597072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:37,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48912 deadline: 1681858597072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:37,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858597073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:37,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858597176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:37,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858597176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:37,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858597179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,320 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/4792272a9c3d44289da62fb2af3de3f3 2023-04-18 22:55:37,330 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/77bdf6c1ad4e4d49a0b6c74c2ddbd5a0 2023-04-18 22:55:37,336 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/9294f9e88d194d5eb56f57b6312dc48d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/9294f9e88d194d5eb56f57b6312dc48d 2023-04-18 22:55:37,340 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/9294f9e88d194d5eb56f57b6312dc48d, entries=150, sequenceid=60, filesize=11.6 K 2023-04-18 22:55:37,340 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/4792272a9c3d44289da62fb2af3de3f3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/4792272a9c3d44289da62fb2af3de3f3 2023-04-18 22:55:37,345 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/4792272a9c3d44289da62fb2af3de3f3, entries=150, sequenceid=60, filesize=11.6 K 2023-04-18 22:55:37,345 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/77bdf6c1ad4e4d49a0b6c74c2ddbd5a0 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/77bdf6c1ad4e4d49a0b6c74c2ddbd5a0 2023-04-18 22:55:37,349 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/77bdf6c1ad4e4d49a0b6c74c2ddbd5a0, entries=150, sequenceid=60, filesize=11.6 K 2023-04-18 22:55:37,350 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~441.19 KB/451776, currentSize=26.84 KB/27480 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 467ms, sequenceid=60, compaction requested=false 2023-04-18 22:55:37,351 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:37,351 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:37,351 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:37,351 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:37,351 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,351 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:37,351 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:37,358 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,358 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:37,358 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:37,358 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:37,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:37,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78d53600[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:37,359 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:37,360 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,361 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,361 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:37,361 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:37,361 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:37,361 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,361 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:37,363 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,363 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:37,363 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:37,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:37,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:37,363 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:37,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:37,366 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:37,366 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:37,366 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:37,366 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:37,366 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:37,367 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:37,368 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,368 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,368 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,369 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,372 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:37,372 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:37,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:37,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:37,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:37,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:37,372 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,373 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,373 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,373 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,373 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,373 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,373 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,372 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:37,373 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:37,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:37,373 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:37,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:37,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:37,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:37,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:37,384 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:55:37,385 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:37,385 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:37,385 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:37,385 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:37,385 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:37,385 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:37,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858537410, max timestamp=1681858537420Num uniques 859; 2023-04-18 22:55:37,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:55:37,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858537410, max timestamp=1681858537420Num uniques 859; 2023-04-18 22:55:37,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:55:37,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858537410, max timestamp=1681858537420Num uniques 859; 2023-04-18 22:55:37,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:55:37,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:37,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858597453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:37,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858597454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:37,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858597455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,473 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:37,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:37,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:37,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:37,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:37,476 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:37,476 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:37,476 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:37,476 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:37,477 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,477 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,478 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:37,478 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:37,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:37,479 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,479 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:37,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:37,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:37,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:37,480 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:37,480 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:37,481 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:37,481 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:37,481 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:37,481 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:37,481 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,481 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:37,481 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:37,482 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:37,483 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,483 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:37,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:37,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3570b689[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:37,484 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:37,484 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,486 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,486 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:37,486 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:37,486 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:37,486 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,486 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:37,488 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,488 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:37,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:37,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:37,488 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:37,489 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:37,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:37,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:37,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:37,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:37,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:37,491 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:37,491 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:37,491 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:37,491 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:37,491 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:37,492 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,492 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:37,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:37,492 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:37,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:37,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:37,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:37,493 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:37,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,499 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,499 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:37,499 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:37,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:37,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:37,499 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:37,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:37,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:37,499 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:37,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:37,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:37,500 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,500 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:37,500 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,500 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:37,500 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,500 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,500 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:37,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858597561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:37,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858597567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:37,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858597567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,600 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:37,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:37,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:37,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:37,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:37,603 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:37,603 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:37,604 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:37,604 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:37,604 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,604 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,607 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:37,607 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:37,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:37,607 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,607 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:37,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:37,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:37,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:37,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:37,608 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:37,608 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:37,608 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:37,609 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:37,609 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:37,609 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:37,609 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:37,609 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:37,609 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:37,609 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:37,609 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:37,609 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,609 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:37,609 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:37,611 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,611 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:37,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:37,612 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,612 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:37,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:37,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:37,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:37,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:37,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74c3822b[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:37,613 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:37,613 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,614 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,614 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:37,614 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:37,614 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:37,614 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,614 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:37,616 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:37,616 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,616 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:37,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:37,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:37,616 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:37,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:37,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:37,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:37,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:37,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:37,618 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:37,618 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:37,618 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:37,618 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:37,618 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:37,621 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,621 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,621 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:37,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:37,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:37,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:37,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:37,622 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:37,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:37,623 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,625 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,625 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,625 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,625 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,625 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,626 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:37,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:37,626 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:37,626 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:37,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:37,626 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:37,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:37,626 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:37,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:37,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:37,627 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:37,627 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:37,727 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:37,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:37,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:37,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:37,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:37,731 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:37,731 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:37,731 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:37,731 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:37,731 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,731 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,732 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,732 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:37,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:37,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:37,733 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,733 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:37,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:37,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:37,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:37,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:37,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:37,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:37,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:37,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:37,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:37,734 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:37,734 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:37,734 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:37,734 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:37,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:37,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:37,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:37,734 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:37,736 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,736 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:37,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:37,736 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,736 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:37,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:37,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:37,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:37,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:37,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d9f3d4e[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:37,738 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:37,738 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,740 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,740 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:37,740 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:37,740 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:37,740 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,740 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:37,741 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:37,741 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:37,741 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,742 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:37,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:37,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:37,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:37,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:37,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:37,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:37,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:37,744 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:37,744 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:37,744 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:37,744 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:37,744 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:37,746 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,746 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:37,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:37,746 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:37,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:37,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:37,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:37,747 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:37,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:37,749 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,753 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,753 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,753 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,753 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,753 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,753 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,753 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,753 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:37,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:37,753 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:37,753 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:37,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:37,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:37,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:37,753 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:37,754 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:37,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:37,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:37,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:37,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858597766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:37,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858597772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:37,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858597776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,792 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/41cafb0d9ccd481ab64d73947afe2288 2023-04-18 22:55:37,854 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:37,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:37,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:37,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:37,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:37,857 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:37,857 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:37,857 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:37,857 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:37,858 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,858 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,859 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,859 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:37,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:37,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:37,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,860 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,860 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:37,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:37,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:37,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:37,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:37,861 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:37,861 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:37,861 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:37,861 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:37,861 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:37,861 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:37,861 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:37,861 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:37,861 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:37,862 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:37,862 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:37,862 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,862 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:37,862 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:37,863 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,863 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:37,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:37,863 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,863 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:37,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:37,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:37,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:37,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:37,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cab6bcb[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:37,865 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:37,865 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,866 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:37,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:37,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:37,866 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,866 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:37,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:37,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:37,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:37,873 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:37,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:37,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:37,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:37,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:37,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:37,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:37,877 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:37,878 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:37,878 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:37,878 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:37,878 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:37,880 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,880 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:37,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,882 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:37,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:37,883 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:37,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,885 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:37,885 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:37,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:37,885 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:37,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:37,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:37,885 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:37,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:37,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:37,885 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,886 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,886 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:37,886 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,886 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,886 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:37,886 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:37,886 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,986 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:37,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:37,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:37,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:37,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:37,990 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:37,990 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:37,990 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:37,990 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:37,991 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,991 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,995 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,995 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:37,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:37,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:37,995 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,995 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:37,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:37,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:37,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:37,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:37,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:37,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:37,997 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:37,997 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:37,997 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:37,997 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:37,997 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:37,999 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:37,999 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:37,999 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:37,999 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:37,999 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:37,999 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:37,999 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:37,999 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:37,999 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:38,002 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:38,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:38,002 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:38,002 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,002 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:38,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:38,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:38,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:38,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c5f5e29[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:38,004 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:38,004 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,005 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,005 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:38,005 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:38,005 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:38,005 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,006 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:38,007 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:38,007 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:38,007 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,007 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:38,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:38,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:38,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:38,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:38,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:38,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:38,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:38,011 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:38,011 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:38,011 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:38,011 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:38,011 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:38,012 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,013 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:38,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:38,013 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:38,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:38,013 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:38,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:38,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:38,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,014 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:38,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,017 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:38,017 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:38,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:38,017 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:38,017 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:38,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:38,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:38,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:38,017 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:38,018 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:38,018 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:38,017 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:38,018 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,018 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:38,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:38,018 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,018 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:38,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858598075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:38,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858598076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:38,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48912 deadline: 1681858598078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:38,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48868 deadline: 1681858598084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:38,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858598086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,118 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:38,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:38,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:38,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:38,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:38,128 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:38,128 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:38,128 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:38,128 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:38,128 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,128 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:38,130 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:38,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:38,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:38,130 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:38,130 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,130 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:38,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:38,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:38,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:38,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:38,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:38,132 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:38,132 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:38,132 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:38,137 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:38,137 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:38,137 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:38,137 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:38,137 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:38,138 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:38,138 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:38,138 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:38,138 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,138 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:38,138 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:38,139 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,139 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:38,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:38,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:38,140 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,140 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:38,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:38,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:38,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:38,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24655f53[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:38,142 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:38,142 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,143 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,143 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:38,143 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:38,143 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,144 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:38,144 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:38,148 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,148 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:38,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:38,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:38,148 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:38,148 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:38,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:38,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:38,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:38,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:38,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:38,150 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:38,150 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:38,150 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:38,150 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:38,151 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:38,152 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,152 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:38,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,152 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:38,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:38,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:38,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:38,153 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:38,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:38,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,155 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:38,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,158 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,158 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:38,158 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:38,159 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,159 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,159 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,159 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,159 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:38,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:38,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:38,159 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:38,159 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:38,159 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:38,159 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:38,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:38,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:38,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:38,203 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/626ff059f61d48018f21fd5a34bf902f 2023-04-18 22:55:38,244 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/c21483fc8a1e4b87b425cea08dcc29f9 2023-04-18 22:55:38,253 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/41cafb0d9ccd481ab64d73947afe2288 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/41cafb0d9ccd481ab64d73947afe2288 2023-04-18 22:55:38,259 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/41cafb0d9ccd481ab64d73947afe2288, entries=200, sequenceid=70, filesize=13.9 K 2023-04-18 22:55:38,260 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:38,260 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/626ff059f61d48018f21fd5a34bf902f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/626ff059f61d48018f21fd5a34bf902f 2023-04-18 22:55:38,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:38,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:38,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:38,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:38,276 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:38,276 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:38,276 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:38,276 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:38,277 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,277 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:38,277 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/626ff059f61d48018f21fd5a34bf902f, entries=150, sequenceid=70, filesize=11.6 K 2023-04-18 22:55:38,278 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,278 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:38,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/c21483fc8a1e4b87b425cea08dcc29f9 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/c21483fc8a1e4b87b425cea08dcc29f9 2023-04-18 22:55:38,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:38,278 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,278 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:38,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:38,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:38,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:38,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:38,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:38,280 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:38,280 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:38,280 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:38,282 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:38,282 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:38,282 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:38,282 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:38,282 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:38,282 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:38,282 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:38,282 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:38,282 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,282 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:38,282 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:38,284 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/c21483fc8a1e4b87b425cea08dcc29f9, entries=150, sequenceid=70, filesize=11.6 K 2023-04-18 22:55:38,285 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 901ms, sequenceid=70, compaction requested=true 2023-04-18 22:55:38,285 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:38,285 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:38,285 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:38,285 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:38,285 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:38,285 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:38,286 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:38,286 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:38,286 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/A is initiating minor compaction (all files) 2023-04-18 22:55:38,286 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/B is initiating minor compaction (all files) 2023-04-18 22:55:38,286 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/A in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:38,286 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/B in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:38,286 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/cac5eb5d37794b9fa5e65ad1466169b8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/9294f9e88d194d5eb56f57b6312dc48d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/41cafb0d9ccd481ab64d73947afe2288] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=37.1 K 2023-04-18 22:55:38,286 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ddf10b00c2854e6d99a7ac61f18737fc, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/4792272a9c3d44289da62fb2af3de3f3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/626ff059f61d48018f21fd5a34bf902f] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=34.8 K 2023-04-18 22:55:38,287 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting cac5eb5d37794b9fa5e65ad1466169b8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=24, earliestPutTs=1681858535904 2023-04-18 22:55:38,287 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting ddf10b00c2854e6d99a7ac61f18737fc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=24, earliestPutTs=1681858535904 2023-04-18 22:55:38,287 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 9294f9e88d194d5eb56f57b6312dc48d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1681858535947 2023-04-18 22:55:38,287 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 4792272a9c3d44289da62fb2af3de3f3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1681858535947 2023-04-18 22:55:38,287 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 41cafb0d9ccd481ab64d73947afe2288, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1681858537067 2023-04-18 22:55:38,287 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 626ff059f61d48018f21fd5a34bf902f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1681858537067 2023-04-18 22:55:38,289 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:38,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:38,289 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,289 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,289 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:38,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:38,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:38,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:38,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:38,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70c473dc[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:38,291 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:38,291 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,292 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,292 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:38,292 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:38,292 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:38,293 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,293 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:38,294 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:38,294 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:38,294 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,294 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:38,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:38,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:38,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:38,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:38,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,300 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#A#compaction#383 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:38,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:38,305 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#B#compaction#384 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:38,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:38,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:38,308 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:38,308 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:38,308 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:38,308 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:38,308 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:38,310 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,310 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,310 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:38,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:38,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:38,311 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,313 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:38,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:38,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,316 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,316 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:38,316 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:38,316 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,316 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,316 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:38,316 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:38,316 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:38,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:38,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:38,316 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:38,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:38,316 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:38,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:38,317 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:38,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:38,317 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:38,417 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:38,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:38,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:38,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:38,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:38,422 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:38,422 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:38,422 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:38,422 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:38,422 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,422 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:38,423 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,423 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:38,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:38,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:38,424 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,424 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:38,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:38,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:38,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:38,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:38,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:38,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:38,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:38,428 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:38,428 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:38,428 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:38,428 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:38,428 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:38,428 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:38,428 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:38,428 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-18 22:55:38,429 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:38,429 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:38,429 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:38,429 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:38,429 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:38,429 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:38,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:38,586 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:38,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:38,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858598606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:38,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858598607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:38,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858598611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:38,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858598712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:38,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858598716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:38,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858598723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,731 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/36b3e895a99f43938df748bfbbcfed10 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/36b3e895a99f43938df748bfbbcfed10 2023-04-18 22:55:38,737 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/bc5bb892c6564300bd65a9b84c5721ad as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/bc5bb892c6564300bd65a9b84c5721ad 2023-04-18 22:55:38,738 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/B of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into 36b3e895a99f43938df748bfbbcfed10(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:38,738 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:38,738 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/B, priority=13, startTime=1681858538285; duration=0sec 2023-04-18 22:55:38,738 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:38,738 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:38,739 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:38,739 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/C is initiating minor compaction (all files) 2023-04-18 22:55:38,739 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/C in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:38,739 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/02c94e12fdcf4d418c13ef2e2736aaa3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/77bdf6c1ad4e4d49a0b6c74c2ddbd5a0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/c21483fc8a1e4b87b425cea08dcc29f9] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=34.8 K 2023-04-18 22:55:38,740 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 02c94e12fdcf4d418c13ef2e2736aaa3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=24, earliestPutTs=1681858535904 2023-04-18 22:55:38,740 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 77bdf6c1ad4e4d49a0b6c74c2ddbd5a0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1681858535947 2023-04-18 22:55:38,740 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting c21483fc8a1e4b87b425cea08dcc29f9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1681858537067 2023-04-18 22:55:38,741 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/A of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into bc5bb892c6564300bd65a9b84c5721ad(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:38,741 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:38,741 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/A, priority=13, startTime=1681858538285; duration=0sec 2023-04-18 22:55:38,741 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:38,745 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#C#compaction#386 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:38,753 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/b96de023d15b4e0d81b041da32744ff7 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/b96de023d15b4e0d81b041da32744ff7 2023-04-18 22:55:38,756 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/C of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into b96de023d15b4e0d81b041da32744ff7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:38,756 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:38,756 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/C, priority=13, startTime=1681858538285; duration=0sec 2023-04-18 22:55:38,756 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:38,838 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/f7fe8449e424433fb937697146eede4d 2023-04-18 22:55:38,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:38,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858598919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:38,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858598931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:38,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:38,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858598931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:39,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858599227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:39,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858599237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:39,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858599239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,248 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/ee20a6d4d307491b88592f1603d9dacd 2023-04-18 22:55:39,257 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/ee766c6a09ba4cc7817bc49f236a8784 2023-04-18 22:55:39,261 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/f7fe8449e424433fb937697146eede4d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f7fe8449e424433fb937697146eede4d 2023-04-18 22:55:39,264 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f7fe8449e424433fb937697146eede4d, entries=150, sequenceid=105, filesize=11.6 K 2023-04-18 22:55:39,265 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/ee20a6d4d307491b88592f1603d9dacd as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ee20a6d4d307491b88592f1603d9dacd 2023-04-18 22:55:39,268 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ee20a6d4d307491b88592f1603d9dacd, entries=150, sequenceid=105, filesize=11.6 K 2023-04-18 22:55:39,268 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/ee766c6a09ba4cc7817bc49f236a8784 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ee766c6a09ba4cc7817bc49f236a8784 2023-04-18 22:55:39,272 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ee766c6a09ba4cc7817bc49f236a8784, entries=150, sequenceid=105, filesize=11.6 K 2023-04-18 22:55:39,272 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 844ms, sequenceid=105, compaction requested=false 2023-04-18 22:55:39,272 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:39,272 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:39,272 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:39,273 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:39,273 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,273 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:39,273 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:39,277 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,277 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:39,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:39,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:39,278 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:39,278 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:39,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:39,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:39,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:39,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:39,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:39,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5716bb3c[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:39,279 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:39,279 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:39,281 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:39,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:39,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:39,281 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:39,281 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:39,281 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:39,281 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,281 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:39,282 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:39,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:39,282 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:39,282 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:39,283 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:39,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:39,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:39,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:39,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:39,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:39,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:39,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:39,285 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:39,285 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:39,285 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:39,285 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:39,285 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:39,288 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:39,289 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:39,289 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:39,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:39,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:39,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:39,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:39,290 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:39,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:39,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:39,291 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:39,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:39,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,293 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:39,293 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:39,293 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,294 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:39,294 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:39,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:39,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:39,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:39,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:39,294 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,294 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:39,294 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:39,294 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:39,294 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:39,294 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:39,294 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:39,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:39,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:39,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:39,394 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:39,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:39,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:39,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:39,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:39,398 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:39,398 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:39,398 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:39,398 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:39,399 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:39,399 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:39,401 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,401 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,401 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:39,403 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:39,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:39,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:39,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:39,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:39,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:39,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:39,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:39,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:39,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:39,406 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:39,406 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:39,406 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:39,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:39,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:39,413 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:39,413 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:39,413 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:55:39,413 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:39,413 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:39,414 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:39,414 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:39,414 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:39,414 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:39,449 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/2745eefd1bbe420ebb23bd13800f01e7 2023-04-18 22:55:39,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:39,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:39,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681858539751, max timestamp=1681858539778Num uniques 859; 2023-04-18 22:55:39,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681858539751, max timestamp=1681858539778Num uniques 859; 2023-04-18 22:55:39,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 22:55:39,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681858539751, max timestamp=1681858539778Num uniques 859; 2023-04-18 22:55:39,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 22:55:39,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-18 22:55:39,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:39,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858599812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:39,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858599819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:39,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858599820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,873 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/837a0d21e295409cba6eb77345de55f3 2023-04-18 22:55:39,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:39,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858599922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:39,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858599927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:39,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858599930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:39,966 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-18 22:55:40,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:40,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48912 deadline: 1681858600090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,106 DEBUG [Thread-2913] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:55:40,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:40,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48868 deadline: 1681858600112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,121 DEBUG [Thread-2909] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:55:40,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:40,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858600133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:40,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858600135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:40,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858600140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,286 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/11b98bc0d9d44dfd83360a22ca3bff56 2023-04-18 22:55:40,290 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/2745eefd1bbe420ebb23bd13800f01e7 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/2745eefd1bbe420ebb23bd13800f01e7 2023-04-18 22:55:40,293 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/2745eefd1bbe420ebb23bd13800f01e7, entries=150, sequenceid=118, filesize=11.6 K 2023-04-18 22:55:40,293 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/837a0d21e295409cba6eb77345de55f3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/837a0d21e295409cba6eb77345de55f3 2023-04-18 22:55:40,297 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/837a0d21e295409cba6eb77345de55f3, entries=150, sequenceid=118, filesize=11.6 K 2023-04-18 22:55:40,297 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/11b98bc0d9d44dfd83360a22ca3bff56 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/11b98bc0d9d44dfd83360a22ca3bff56 2023-04-18 22:55:40,300 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/11b98bc0d9d44dfd83360a22ca3bff56, entries=150, sequenceid=118, filesize=11.6 K 2023-04-18 22:55:40,301 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 888ms, sequenceid=118, compaction requested=true 2023-04-18 22:55:40,301 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:40,301 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:40,301 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:40,301 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:40,301 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,301 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:40,301 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:40,303 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,303 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:40,303 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:40,303 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:40,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:40,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:40,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:40,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@610e09e1[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:40,304 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:40,304 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:40,306 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:40,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:40,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:40,306 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:40,306 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:40,306 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:40,306 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,306 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:40,308 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:40,308 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,308 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:40,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:40,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:40,308 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:40,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:40,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:40,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:40,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:40,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:40,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:40,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:40,310 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:40,310 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:40,310 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:40,310 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:40,310 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:40,311 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:40,311 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:40,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:40,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:40,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:40,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:40,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:40,312 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:40,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:40,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:40,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:40,312 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:40,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:40,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:40,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:40,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,313 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:40,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:40,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,315 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,315 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:40,315 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:40,315 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,315 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:40,315 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:40,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:40,315 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:40,315 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:40,315 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:40,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:40,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:40,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:40,315 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:40,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:40,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:40,316 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:40,316 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:40,416 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:40,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:40,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:40,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:40,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:40,420 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:40,420 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:40,420 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:40,420 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:40,422 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:40,422 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:40,425 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:40,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:40,425 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:40,426 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:40,426 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:40,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:40,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:40,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:40,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:40,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:40,427 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:40,427 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:40,427 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:40,427 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:40,427 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:40,427 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:40,427 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:40,427 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-18 22:55:40,427 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:40,428 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:40,428 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:40,428 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:40,428 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:40,428 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:40,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:40,451 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:40,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:40,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858600460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:40,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858600463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:40,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858600468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:40,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858600569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:40,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858600576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:40,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858600577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:40,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858600777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:40,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858600781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:40,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858600786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,843 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/e103d911f8b54a51b341926e9d1ec3ed 2023-04-18 22:55:40,858 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/ea373cf7dc3d4535a7007a0d37f7f27a 2023-04-18 22:55:40,870 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/ce11bedbcede4a429dd6dd873a48dddd 2023-04-18 22:55:40,874 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/e103d911f8b54a51b341926e9d1ec3ed as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/e103d911f8b54a51b341926e9d1ec3ed 2023-04-18 22:55:40,878 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/e103d911f8b54a51b341926e9d1ec3ed, entries=150, sequenceid=153, filesize=11.8 K 2023-04-18 22:55:40,879 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/ea373cf7dc3d4535a7007a0d37f7f27a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ea373cf7dc3d4535a7007a0d37f7f27a 2023-04-18 22:55:40,884 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ea373cf7dc3d4535a7007a0d37f7f27a, entries=150, sequenceid=153, filesize=11.8 K 2023-04-18 22:55:40,885 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/ce11bedbcede4a429dd6dd873a48dddd as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ce11bedbcede4a429dd6dd873a48dddd 2023-04-18 22:55:40,889 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ce11bedbcede4a429dd6dd873a48dddd, entries=150, sequenceid=153, filesize=11.8 K 2023-04-18 22:55:40,890 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 463ms, sequenceid=153, compaction requested=true 2023-04-18 22:55:40,890 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:40,890 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:40,890 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:40,890 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:40,890 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,892 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:40,892 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:40,893 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,893 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:40,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:40,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:40,894 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:40,894 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:40,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:40,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:40,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:40,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:40,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:40,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f1f4d54[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:40,895 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:40,895 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:40,896 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:40,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:40,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:40,897 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:40,897 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:40,897 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:40,897 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,897 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:40,898 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,898 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:40,898 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:40,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:40,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:40,898 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:40,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:40,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:40,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:40,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:40,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:40,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:40,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:40,902 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:40,902 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:40,902 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:40,902 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:40,902 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:40,909 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:40,909 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:40,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:40,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:40,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:40,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:40,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:40,909 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:40,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:40,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:40,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:40,910 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:40,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:40,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:40,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:40,912 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:40,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:40,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,915 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:40,915 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:40,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:40,915 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:40,915 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:40,915 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:40,915 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:40,915 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:40,915 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:40,915 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:40,915 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:40,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:40,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:40,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:40,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:40,916 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:40,916 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:40,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:41,016 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:41,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:41,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:41,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:41,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:41,020 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:41,021 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:41,021 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:41,021 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:41,021 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:41,021 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:41,022 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:41,023 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:41,023 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,023 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:41,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:41,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:41,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:41,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:41,024 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:41,024 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:41,024 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:41,025 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:41,025 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:41,025 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:41,025 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:41,025 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:55:41,025 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:41,025 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:41,025 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:41,025 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:41,025 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:41,025 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:41,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:41,090 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:41,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858541128, max timestamp=1681858541142Num uniques 859; 2023-04-18 22:55:41,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:55:41,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858541128, max timestamp=1681858541142Num uniques 859; 2023-04-18 22:55:41,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:55:41,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858541128, max timestamp=1681858541142Num uniques 859; 2023-04-18 22:55:41,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:55:41,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:41,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858601176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:41,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858601177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:41,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858601179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:41,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858601281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:41,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858601286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:41,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858601288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,438 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/42a9a4c9d20f465785b00048ea87db77 2023-04-18 22:55:41,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:41,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858601488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:41,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858601493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:41,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858601493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:41,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858601792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:41,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858601802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:41,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858601803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,856 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/9cffeb0751934ecfaff7f4f753ca6fa9 2023-04-18 22:55:41,869 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/12ffcf80ada64e2386ad9c020a94738a 2023-04-18 22:55:41,873 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/42a9a4c9d20f465785b00048ea87db77 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/42a9a4c9d20f465785b00048ea87db77 2023-04-18 22:55:41,879 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/42a9a4c9d20f465785b00048ea87db77, entries=150, sequenceid=163, filesize=11.8 K 2023-04-18 22:55:41,879 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/9cffeb0751934ecfaff7f4f753ca6fa9 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/9cffeb0751934ecfaff7f4f753ca6fa9 2023-04-18 22:55:41,883 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/9cffeb0751934ecfaff7f4f753ca6fa9, entries=150, sequenceid=163, filesize=11.8 K 2023-04-18 22:55:41,883 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/12ffcf80ada64e2386ad9c020a94738a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/12ffcf80ada64e2386ad9c020a94738a 2023-04-18 22:55:41,888 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/12ffcf80ada64e2386ad9c020a94738a, entries=150, sequenceid=163, filesize=11.8 K 2023-04-18 22:55:41,889 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 864ms, sequenceid=163, compaction requested=true 2023-04-18 22:55:41,889 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:41,889 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:41,889 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:41,889 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:41,889 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,889 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:41,889 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:41,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,890 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:41,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:41,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:41,891 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:41,891 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:41,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:41,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:41,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:41,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:41,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:41,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c1689f8[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:41,893 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:41,893 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:41,894 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:41,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:41,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:41,894 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:41,894 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:41,894 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:41,894 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,894 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:41,898 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,898 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:41,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:41,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:41,898 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:41,899 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:41,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:41,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:41,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:41,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:41,901 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:41,901 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:41,901 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:41,901 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:41,901 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:41,902 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:41,902 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:41,902 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:41,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:41,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:41,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:41,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:41,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:41,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:41,903 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:41,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:41,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:41,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:41,904 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,906 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,906 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:41,906 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:41,906 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:41,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:41,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:41,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:41,906 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:41,906 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:41,906 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:41,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:41,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:41,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:41,906 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:41,907 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:41,907 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:41,907 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:41,906 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:42,007 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:42,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:42,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:42,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:42,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:42,011 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:42,011 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:42,011 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:42,011 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:42,012 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:42,012 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:42,013 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:42,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:42,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:42,013 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:42,014 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,014 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:42,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:42,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:42,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:42,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:42,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:42,015 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:42,015 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:42,015 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:42,015 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:42,015 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:42,015 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:42,015 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:42,019 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-18 22:55:42,019 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:42,019 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:42,019 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:42,019 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:42,019 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:42,019 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:42,029 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/a76db7b1705645e08e06e16ad44ff888 2023-04-18 22:55:42,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:42,306 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:42,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:42,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858602329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:42,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858602336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:42,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858602337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:42,447 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/6bb7f7beff1348fbaab679f3529f97a8 2023-04-18 22:55:42,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858602443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:42,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858602451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:42,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858602452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,460 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/608d85f0dfe14794a3c1c64197297827 2023-04-18 22:55:42,463 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/a76db7b1705645e08e06e16ad44ff888 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/a76db7b1705645e08e06e16ad44ff888 2023-04-18 22:55:42,468 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/a76db7b1705645e08e06e16ad44ff888, entries=150, sequenceid=198, filesize=11.8 K 2023-04-18 22:55:42,469 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/6bb7f7beff1348fbaab679f3529f97a8 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/6bb7f7beff1348fbaab679f3529f97a8 2023-04-18 22:55:42,474 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/6bb7f7beff1348fbaab679f3529f97a8, entries=150, sequenceid=198, filesize=11.8 K 2023-04-18 22:55:42,475 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/608d85f0dfe14794a3c1c64197297827 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/608d85f0dfe14794a3c1c64197297827 2023-04-18 22:55:42,478 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/608d85f0dfe14794a3c1c64197297827, entries=150, sequenceid=198, filesize=11.8 K 2023-04-18 22:55:42,478 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 460ms, sequenceid=198, compaction requested=true 2023-04-18 22:55:42,478 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:42,478 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:42,479 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:42,479 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:42,479 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,479 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:42,479 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:42,481 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,481 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:42,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:42,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:42,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:42,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:42,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:42,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e3b1a1d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:42,483 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:42,483 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:42,484 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:42,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:42,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:42,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:42,485 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:42,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:42,486 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:42,487 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:42,487 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:42,488 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:42,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:42,490 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:42,490 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:42,490 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:42,490 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:42,490 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:42,492 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:42,492 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:42,493 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:42,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:42,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:42,493 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:42,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:42,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,495 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,498 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,498 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:42,498 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:42,498 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:42,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:42,498 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:42,499 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:42,499 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:42,498 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:42,499 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:42,499 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:42,498 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:42,499 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:42,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:42,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:42,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:42,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:42,599 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:42,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:42,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:42,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:42,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:42,603 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:42,603 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:42,603 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:42,603 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:42,603 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:42,603 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:42,604 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:42,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:42,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:42,604 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:42,605 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,605 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:42,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:42,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:42,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:42,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:42,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:42,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:42,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:42,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:42,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:42,606 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:42,606 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:42,606 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:55:42,606 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:42,606 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:42,607 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:42,607 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:42,607 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:42,607 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:42,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:42,652 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:42,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858542698, max timestamp=1681858542707Num uniques 859; 2023-04-18 22:55:42,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858542698, max timestamp=1681858542707Num uniques 859; 2023-04-18 22:55:42,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858542698, max timestamp=1681858542707Num uniques 859; 2023-04-18 22:55:42,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:55:42,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:55:42,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:55:42,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:42,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858602762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:42,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858602763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:42,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858602764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:42,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858602883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:42,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858602883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:42,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:42,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858602883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,014 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/f498d7b1615744ff817ba7d708082d8d 2023-04-18 22:55:43,023 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/924df5d0684d4d658bc3e1c645f3cc37 2023-04-18 22:55:43,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:43,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858603093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:43,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858603093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:43,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858603094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:43,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48892 deadline: 1681858603403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:43,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48880 deadline: 1681858603404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:43,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858603405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,431 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/fd089b48bdf049a491d6a3da1c6076a7 2023-04-18 22:55:43,435 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/f498d7b1615744ff817ba7d708082d8d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f498d7b1615744ff817ba7d708082d8d 2023-04-18 22:55:43,438 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f498d7b1615744ff817ba7d708082d8d, entries=150, sequenceid=208, filesize=11.8 K 2023-04-18 22:55:43,439 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/924df5d0684d4d658bc3e1c645f3cc37 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/924df5d0684d4d658bc3e1c645f3cc37 2023-04-18 22:55:43,450 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/924df5d0684d4d658bc3e1c645f3cc37, entries=150, sequenceid=208, filesize=11.8 K 2023-04-18 22:55:43,451 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/fd089b48bdf049a491d6a3da1c6076a7 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/fd089b48bdf049a491d6a3da1c6076a7 2023-04-18 22:55:43,457 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/fd089b48bdf049a491d6a3da1c6076a7, entries=150, sequenceid=208, filesize=11.8 K 2023-04-18 22:55:43,458 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 852ms, sequenceid=208, compaction requested=true 2023-04-18 22:55:43,458 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:43,459 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:43,459 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:43,459 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:43,459 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,459 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:43,459 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:43,460 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,460 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:43,461 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:43,461 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:43,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:43,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4feb413a[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:43,463 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:43,463 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:43,464 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:43,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:43,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:43,464 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:43,464 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:43,464 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,465 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:43,464 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:43,467 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:43,467 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:43,468 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:43,468 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:43,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:43,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:43,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:43,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:43,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:43,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:43,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:43,471 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:43,471 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:43,471 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:43,471 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:43,471 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:43,473 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:43,473 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:43,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:43,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:43,473 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:43,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:43,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:43,474 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,474 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:43,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:43,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:43,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:43,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:43,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:43,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:43,477 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:43,477 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:43,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:43,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:43,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:43,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:43,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:43,477 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:43,477 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:43,477 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:43,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:43,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:43,477 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,479 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:43,479 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:43,479 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,479 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:43,479 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:43,479 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:43,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:43,577 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:43,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:43,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:43,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:43,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:43,584 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:43,584 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:43,584 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:43,584 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:43,584 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:43,585 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:43,586 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,586 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:43,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:43,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:43,586 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,586 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:43,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:43,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:43,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:43,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:43,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:43,588 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:43,588 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:43,588 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:43,588 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:43,588 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:43,589 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:43,589 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:43,589 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-18 22:55:43,589 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:43,589 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:43,589 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:43,589 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:43,589 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:43,589 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:43,597 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/b515a99a50a745ee97cee9169a89f3e1 2023-04-18 22:55:43,606 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/30baad6749cc43769c6756cbb7b518fd 2023-04-18 22:55:43,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:43,922 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:43,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:43,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858603935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:43,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48956 deadline: 1681858603935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:43,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:43,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858603940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,015 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/8376abd9522f4a8d95bc9d35954c55d8 2023-04-18 22:55:44,019 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/b515a99a50a745ee97cee9169a89f3e1 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/b515a99a50a745ee97cee9169a89f3e1 2023-04-18 22:55:44,022 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/b515a99a50a745ee97cee9169a89f3e1, entries=150, sequenceid=245, filesize=11.8 K 2023-04-18 22:55:44,023 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/30baad6749cc43769c6756cbb7b518fd as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/30baad6749cc43769c6756cbb7b518fd 2023-04-18 22:55:44,026 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/30baad6749cc43769c6756cbb7b518fd, entries=150, sequenceid=245, filesize=11.8 K 2023-04-18 22:55:44,027 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/8376abd9522f4a8d95bc9d35954c55d8 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/8376abd9522f4a8d95bc9d35954c55d8 2023-04-18 22:55:44,030 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/8376abd9522f4a8d95bc9d35954c55d8, entries=150, sequenceid=245, filesize=11.8 K 2023-04-18 22:55:44,031 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 442ms, sequenceid=245, compaction requested=true 2023-04-18 22:55:44,031 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:44,031 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:44,031 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:44,031 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:44,031 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,031 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:44,031 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:44,034 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,034 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:44,034 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,034 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:44,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:44,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:44,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:44,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a0bd07e[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:44,035 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:44,036 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,037 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,037 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:44,037 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:44,037 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:44,037 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,037 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:44,039 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:44,039 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,039 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:44,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:44,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:44,039 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:44,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:44,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:44,043 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:44,043 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:44,043 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:44,043 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:44,043 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:44,048 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,048 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:44,048 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:44,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:44,048 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,049 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:44,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,051 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,051 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,051 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,051 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,051 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:44,052 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:44,051 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:44,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:44,051 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:44,052 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,052 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:44,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:44,052 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:44,052 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:44,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:44,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:44,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:44,053 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-18 22:55:44,054 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:44,054 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:44,054 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:44,054 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:44,054 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:44,054 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:44,063 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/3410d5d08378447096a3b790128fbc53 2023-04-18 22:55:44,072 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/68e0d02d45ba4d5889d97d5336fb1c17 2023-04-18 22:55:44,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:44,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858604107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:44,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858604110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:44,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858604118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:44,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48912 deadline: 1681858604134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,141 DEBUG [Thread-2913] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8187 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:55:44,152 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:44,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:44,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:44,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:44,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:44,156 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:44,156 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:44,156 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:44,156 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:44,156 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,156 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,157 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:44,157 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:44,158 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,158 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:44,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:44,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:44,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:44,159 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:44,159 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:44,159 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:44,159 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:44,159 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:44,159 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:44,159 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:44,159 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:44,160 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:44,160 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:44,160 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:44,160 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,160 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:44,160 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:44,161 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,161 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:44,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:44,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:44,162 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,162 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:44,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48868 deadline: 1681858604154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:44,163 DEBUG [Thread-2909] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8210 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., hostname=jenkins-hbase4.apache.org,35971,1681858424869, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-18 22:55:44,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:44,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:44,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:44,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52dadb8e[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:44,165 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:44,165 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,167 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,167 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:44,167 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:44,167 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:44,168 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,168 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:44,169 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:44,169 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,169 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:44,169 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:44,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:44,171 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:44,171 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:44,171 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:44,171 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:44,171 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:44,173 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,173 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:44,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,173 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:44,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:44,173 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:44,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:44,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:44,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:44,174 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:44,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,177 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,177 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:44,177 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:44,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:44,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:44,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:44,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:44,177 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:44,177 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,178 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,178 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:44,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:44,178 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:44,178 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:44,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:44,178 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,178 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,178 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:44,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858604218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:44,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858604218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:44,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858604228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,278 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:44,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:44,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:44,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:44,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:44,284 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:44,284 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:44,284 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:44,284 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:44,284 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,284 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,286 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:44,286 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:44,287 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,287 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:44,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:44,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:44,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:44,288 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:44,288 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:44,288 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:44,288 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:44,288 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:44,288 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:44,288 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:44,288 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:44,288 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:44,289 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:44,289 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:44,289 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,289 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:44,289 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:44,290 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,290 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:44,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:44,291 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,291 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:44,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:44,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:44,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:44,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:44,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5437c83a[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:44,292 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:44,292 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,293 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,293 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:44,293 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:44,293 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:44,294 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,294 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:44,295 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,295 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:44,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:44,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:44,295 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:44,295 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:44,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:44,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:44,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:44,298 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:44,298 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:44,298 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:44,298 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:44,298 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:44,300 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,300 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:44,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,300 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:44,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:44,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:44,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,301 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:44,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,302 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:44,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,304 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,304 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,304 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,304 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:44,304 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:44,304 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:44,304 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:44,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:44,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:44,304 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,304 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,304 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:44,304 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:44,305 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:44,405 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:44,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:44,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:44,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:44,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:44,408 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:44,408 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:44,408 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:44,408 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:44,409 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,409 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,410 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:44,410 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:44,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:44,410 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,410 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:44,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:44,411 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:44,411 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:44,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:44,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:44,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:44,412 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:44,412 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:44,412 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:44,412 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:44,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:44,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:44,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:44,412 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:44,413 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,413 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:44,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:44,414 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,414 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:44,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:44,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:44,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:44,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:44,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41c0b117[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:44,415 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:44,415 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,416 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,416 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:44,416 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:44,416 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:44,416 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,416 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:44,418 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,419 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:44,419 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:44,419 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:44,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:44,421 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:44,421 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:44,421 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:44,421 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:44,421 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:44,423 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,423 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:44,423 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:44,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:44,423 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,424 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,429 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,429 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,429 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,429 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:44,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:44,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:44,429 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:44,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:44,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:44,429 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:44,429 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,430 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,430 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:44,430 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,430 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:44,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:44,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:44,430 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:44,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:44,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858604426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:44,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858604426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:44,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858604436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,481 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/008a4afef0db4c39af0c98c583016e3f 2023-04-18 22:55:44,485 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/3410d5d08378447096a3b790128fbc53 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3410d5d08378447096a3b790128fbc53 2023-04-18 22:55:44,488 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3410d5d08378447096a3b790128fbc53, entries=250, sequenceid=257, filesize=16.5 K 2023-04-18 22:55:44,489 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/68e0d02d45ba4d5889d97d5336fb1c17 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/68e0d02d45ba4d5889d97d5336fb1c17 2023-04-18 22:55:44,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/68e0d02d45ba4d5889d97d5336fb1c17, entries=150, sequenceid=257, filesize=11.8 K 2023-04-18 22:55:44,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/008a4afef0db4c39af0c98c583016e3f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/008a4afef0db4c39af0c98c583016e3f 2023-04-18 22:55:44,496 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/008a4afef0db4c39af0c98c583016e3f, entries=150, sequenceid=257, filesize=11.8 K 2023-04-18 22:55:44,497 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 444ms, sequenceid=257, compaction requested=true 2023-04-18 22:55:44,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:44,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:44,497 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-18 22:55:44,497 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-18 22:55:44,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:44,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:44,499 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 112807 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-18 22:55:44,499 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/A is initiating minor compaction (all files) 2023-04-18 22:55:44,499 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 108027 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-18 22:55:44,499 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/A in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:44,500 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/B is initiating minor compaction (all files) 2023-04-18 22:55:44,500 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/B in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:44,500 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/bc5bb892c6564300bd65a9b84c5721ad, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f7fe8449e424433fb937697146eede4d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/2745eefd1bbe420ebb23bd13800f01e7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/e103d911f8b54a51b341926e9d1ec3ed, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/42a9a4c9d20f465785b00048ea87db77, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/a76db7b1705645e08e06e16ad44ff888, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f498d7b1615744ff817ba7d708082d8d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/b515a99a50a745ee97cee9169a89f3e1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3410d5d08378447096a3b790128fbc53] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=110.2 K 2023-04-18 22:55:44,500 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/36b3e895a99f43938df748bfbbcfed10, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ee20a6d4d307491b88592f1603d9dacd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/837a0d21e295409cba6eb77345de55f3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ea373cf7dc3d4535a7007a0d37f7f27a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/9cffeb0751934ecfaff7f4f753ca6fa9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/6bb7f7beff1348fbaab679f3529f97a8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/924df5d0684d4d658bc3e1c645f3cc37, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/30baad6749cc43769c6756cbb7b518fd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/68e0d02d45ba4d5889d97d5336fb1c17] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=105.5 K 2023-04-18 22:55:44,500 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting bc5bb892c6564300bd65a9b84c5721ad, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1681858537067 2023-04-18 22:55:44,500 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 36b3e895a99f43938df748bfbbcfed10, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1681858537067 2023-04-18 22:55:44,500 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting f7fe8449e424433fb937697146eede4d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1681858537445 2023-04-18 22:55:44,500 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting ee20a6d4d307491b88592f1603d9dacd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1681858537445 2023-04-18 22:55:44,501 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 2745eefd1bbe420ebb23bd13800f01e7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1681858538595 2023-04-18 22:55:44,501 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 837a0d21e295409cba6eb77345de55f3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1681858538595 2023-04-18 22:55:44,501 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting e103d911f8b54a51b341926e9d1ec3ed, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1681858539808 2023-04-18 22:55:44,501 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting ea373cf7dc3d4535a7007a0d37f7f27a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1681858539808 2023-04-18 22:55:44,501 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 42a9a4c9d20f465785b00048ea87db77, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681858540459 2023-04-18 22:55:44,501 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 9cffeb0751934ecfaff7f4f753ca6fa9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681858540459 2023-04-18 22:55:44,502 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting a76db7b1705645e08e06e16ad44ff888, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681858541167 2023-04-18 22:55:44,502 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 6bb7f7beff1348fbaab679f3529f97a8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681858541167 2023-04-18 22:55:44,502 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting f498d7b1615744ff817ba7d708082d8d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681858542328 2023-04-18 22:55:44,502 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 924df5d0684d4d658bc3e1c645f3cc37, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681858542328 2023-04-18 22:55:44,502 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting b515a99a50a745ee97cee9169a89f3e1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1681858542761 2023-04-18 22:55:44,502 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 30baad6749cc43769c6756cbb7b518fd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1681858542761 2023-04-18 22:55:44,502 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 3410d5d08378447096a3b790128fbc53, keycount=250, bloomtype=ROW, size=16.5 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681858543923 2023-04-18 22:55:44,503 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 68e0d02d45ba4d5889d97d5336fb1c17, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681858543939 2023-04-18 22:55:44,514 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#A#compaction#410 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:44,516 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#B#compaction#411 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:44,523 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/1cab0c1cf48e46ada1c995d18294725f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/1cab0c1cf48e46ada1c995d18294725f 2023-04-18 22:55:44,526 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/B of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into 1cab0c1cf48e46ada1c995d18294725f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:44,526 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:44,526 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/B, priority=7, startTime=1681858544497; duration=0sec 2023-04-18 22:55:44,526 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:44,526 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-18 22:55:44,528 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 108027 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-18 22:55:44,528 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/C is initiating minor compaction (all files) 2023-04-18 22:55:44,528 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/C in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:44,529 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/b96de023d15b4e0d81b041da32744ff7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ee766c6a09ba4cc7817bc49f236a8784, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/11b98bc0d9d44dfd83360a22ca3bff56, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ce11bedbcede4a429dd6dd873a48dddd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/12ffcf80ada64e2386ad9c020a94738a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/608d85f0dfe14794a3c1c64197297827, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/fd089b48bdf049a491d6a3da1c6076a7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/8376abd9522f4a8d95bc9d35954c55d8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/008a4afef0db4c39af0c98c583016e3f] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=105.5 K 2023-04-18 22:55:44,529 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting b96de023d15b4e0d81b041da32744ff7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1681858537067 2023-04-18 22:55:44,529 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting ee766c6a09ba4cc7817bc49f236a8784, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1681858537445 2023-04-18 22:55:44,529 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 11b98bc0d9d44dfd83360a22ca3bff56, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1681858538595 2023-04-18 22:55:44,529 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting ce11bedbcede4a429dd6dd873a48dddd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1681858539808 2023-04-18 22:55:44,530 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 12ffcf80ada64e2386ad9c020a94738a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681858540459 2023-04-18 22:55:44,530 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 608d85f0dfe14794a3c1c64197297827, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681858541167 2023-04-18 22:55:44,530 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting fd089b48bdf049a491d6a3da1c6076a7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681858542328 2023-04-18 22:55:44,530 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:44,530 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 8376abd9522f4a8d95bc9d35954c55d8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1681858542761 2023-04-18 22:55:44,531 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 008a4afef0db4c39af0c98c583016e3f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681858543939 2023-04-18 22:55:44,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:44,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:44,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:44,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:44,533 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:44,533 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:44,533 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:44,534 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:44,534 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,534 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,536 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:44,536 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:44,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:44,537 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,537 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:44,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:44,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:44,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:44,538 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:44,538 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:44,538 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:44,538 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:44,538 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:44,538 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:44,538 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:44,538 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 22:55:44,538 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:44,539 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:44,539 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:44,539 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:44,539 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:44,539 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:44,543 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#C#compaction#412 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:44,553 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/9e7a412eee164eeab85bafb5187ce5b6 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/9e7a412eee164eeab85bafb5187ce5b6 2023-04-18 22:55:44,556 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/C of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into 9e7a412eee164eeab85bafb5187ce5b6(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:44,556 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:44,556 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/C, priority=7, startTime=1681858544497; duration=0sec 2023-04-18 22:55:44,556 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:44,562 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/6a11d7f1c2c9463db78e6ab6a31c1ef0 2023-04-18 22:55:44,742 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:44,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:44,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:44,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858604763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:44,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858604763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:44,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858604764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:44,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858604873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:44,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858604874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:44,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858604875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:44,925 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/0dc916ebc0264289b62f3cad08ccbb4d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/0dc916ebc0264289b62f3cad08ccbb4d 2023-04-18 22:55:44,930 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/A of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into 0dc916ebc0264289b62f3cad08ccbb4d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:44,930 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:44,930 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/A, priority=7, startTime=1681858544497; duration=0sec 2023-04-18 22:55:44,930 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:44,974 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/9c658c50c4784187b192e4ecbc0fd222 2023-04-18 22:55:44,989 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/53f9ab31274d402a93996c5cf7db1190 2023-04-18 22:55:44,996 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/6a11d7f1c2c9463db78e6ab6a31c1ef0 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/6a11d7f1c2c9463db78e6ab6a31c1ef0 2023-04-18 22:55:45,001 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/6a11d7f1c2c9463db78e6ab6a31c1ef0, entries=150, sequenceid=278, filesize=11.9 K 2023-04-18 22:55:45,002 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/9c658c50c4784187b192e4ecbc0fd222 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/9c658c50c4784187b192e4ecbc0fd222 2023-04-18 22:55:45,006 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/9c658c50c4784187b192e4ecbc0fd222, entries=150, sequenceid=278, filesize=11.9 K 2023-04-18 22:55:45,007 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/53f9ab31274d402a93996c5cf7db1190 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/53f9ab31274d402a93996c5cf7db1190 2023-04-18 22:55:45,010 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/53f9ab31274d402a93996c5cf7db1190, entries=150, sequenceid=278, filesize=11.9 K 2023-04-18 22:55:45,011 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 473ms, sequenceid=278, compaction requested=false 2023-04-18 22:55:45,011 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:45,011 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:45,013 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,013 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,013 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,014 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:45,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45cc1171[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:45,016 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:45,017 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,018 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:45,018 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,018 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:45,019 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:45,021 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:45,022 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:45,022 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:45,022 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:45,022 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:45,024 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,024 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,024 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,024 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,026 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,028 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,028 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,028 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,028 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,028 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,028 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,028 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:45,028 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:45,028 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:45,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:45,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:45,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:45,028 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:45,028 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:45,028 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:45,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:45,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:45,081 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 22:55:45,081 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:45,082 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:45,082 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:45,082 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:45,082 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:45,082 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:45,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:45,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858605112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:45,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858605113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:45,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858605117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,128 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:45,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:45,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:45,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:45,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:45,131 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:45,131 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:45,131 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:45,131 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:45,132 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,132 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,133 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:45,133 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:45,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:45,133 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,133 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:45,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:45,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:45,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:45,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:45,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:45,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:45,135 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:45,135 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:45,135 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:45,135 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,135 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:45,135 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:45,135 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,135 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:45,135 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:45,135 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,135 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:45,135 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:45,136 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:45,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@414ea4ed[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:45,138 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:45,138 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,140 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,141 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:45,141 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:45,141 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:45,141 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,141 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:45,142 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:45,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:45,146 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:45,146 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:45,146 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:45,146 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:45,146 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:45,148 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,148 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,148 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:45,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:45,148 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,149 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,152 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:45,152 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:45,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:45,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:45,152 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:45,152 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:45,152 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,152 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:45,152 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:45,152 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,152 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:45,153 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:45,153 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,153 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:45,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858605219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:45,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858605220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:45,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858605220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,253 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:45,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:45,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:45,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:45,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:45,256 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:45,256 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:45,256 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:45,256 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:45,256 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,256 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,259 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,259 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:45,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:45,259 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,259 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:45,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:45,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:45,260 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,260 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:45,260 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:45,260 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,261 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:45,261 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:45,261 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,261 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:45,261 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:45,263 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,263 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,264 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,264 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:45,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bd2c2e6[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:45,265 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:45,265 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,266 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:45,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:45,266 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:45,266 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,267 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:45,268 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:45,268 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,268 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,268 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:45,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:45,270 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:45,270 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:45,270 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:45,270 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:45,270 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:45,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,273 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:45,273 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:45,274 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:45,276 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:45,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:45,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:45,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:45,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:45,277 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,277 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,277 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:45,277 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,277 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:45,277 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,277 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:45,377 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:45,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:45,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:45,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:45,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:45,381 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:45,381 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:45,381 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:45,381 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:45,381 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,381 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,386 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,386 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,386 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:45,386 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:45,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:45,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:45,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:45,388 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,388 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:45,389 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:45,389 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,389 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:45,389 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:45,389 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,389 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:45,389 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:45,391 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,391 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,391 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,391 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fde9682[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:45,393 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:45,394 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,396 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,396 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:45,396 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:45,396 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:45,397 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,397 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:45,399 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,399 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,399 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:45,399 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:45,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:45,403 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:45,403 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:45,403 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:45,403 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:45,403 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:45,406 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,406 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:45,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:45,406 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,407 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,408 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,410 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,410 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,410 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,410 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,410 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,410 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:45,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:45,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:45,410 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,410 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,410 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:45,410 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:45,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:45,411 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:45,411 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:45,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:45,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:45,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858605425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:45,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858605425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:45,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858605426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,490 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/8123afaaa41c41e9a110a27418c52559 2023-04-18 22:55:45,511 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:45,511 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/4f9fbe176217467b944e6730bf670a78 2023-04-18 22:55:45,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:45,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:45,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:45,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:45,515 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:45,515 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:45,515 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:45,515 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:45,515 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,515 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,518 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:45,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:45,518 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,520 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,520 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:45,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:45,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:45,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:45,521 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:45,521 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:45,521 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:45,528 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:45,528 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:45,528 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,528 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:45,528 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:45,528 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,528 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:45,528 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:45,528 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,529 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:45,529 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:45,531 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,531 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,532 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,532 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:45,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a3c5591[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:45,535 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:45,535 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,536 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,536 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:45,536 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:45,536 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:45,537 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,537 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:45,538 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,538 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:45,538 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:45,538 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:45,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:45,541 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:45,541 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:45,541 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:45,542 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:45,542 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:45,544 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,544 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,544 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:45,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:45,545 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,547 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,550 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,550 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,550 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:45,550 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:45,550 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:45,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:45,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:45,550 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:45,551 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:45,551 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,551 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,551 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:45,551 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:45,552 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:45,653 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:45,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:45,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:45,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:45,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:45,657 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:45,657 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:45,657 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:45,657 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:45,657 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,657 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,659 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,659 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:45,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:45,659 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,659 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:45,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:45,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:45,660 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:45,660 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:45,660 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:45,660 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:45,660 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:45,661 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,661 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:45,663 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:45,663 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,663 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:45,663 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:45,663 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,663 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:45,663 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:45,665 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,665 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,665 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,665 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:45,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:45,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e61f9f7[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:45,667 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:45,667 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,669 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,669 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:45,669 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:45,669 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,670 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:45,669 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:45,671 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:45,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:45,673 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:45,673 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:45,673 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:45,673 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:45,673 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:45,675 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,675 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,675 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:45,675 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,676 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,679 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:45,679 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:45,679 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:45,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:45,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:45,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:45,679 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,679 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,679 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,679 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:45,679 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:45,679 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:45,679 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:45,679 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,680 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:45,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858605731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:45,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858605732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:45,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858605732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,779 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:45,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:45,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:45,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:45,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:45,783 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:45,783 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:45,783 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:45,783 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:45,783 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,783 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,787 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,787 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:45,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:45,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:45,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,788 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,788 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:45,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:45,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:45,790 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:45,790 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:45,790 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:45,790 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:45,790 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:45,790 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,790 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:45,790 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:45,790 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,790 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:45,790 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:45,790 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,790 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:45,790 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:45,795 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,795 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,796 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,796 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:45,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:45,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4399c12f[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:45,797 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:45,797 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,798 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,798 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:45,798 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:45,799 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,799 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:45,799 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:45,801 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,801 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:45,801 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,801 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:45,803 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:45,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:45,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:45,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:45,803 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:45,805 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,805 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,805 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,805 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,809 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,809 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:45,809 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:45,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:45,809 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:45,809 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:45,809 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:45,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:45,809 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:45,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:45,809 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,810 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,810 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,810 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:45,810 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:45,910 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:45,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:45,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:45,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:45,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:45,913 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:45,913 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:45,913 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:45,913 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:45,913 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,913 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,915 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:45,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:45,915 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,915 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,915 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:45,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:45,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:45,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:45,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:45,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:45,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:45,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:45,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:45,916 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,916 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:45,916 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:45,916 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:45,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:45,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:45,916 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:45,919 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,919 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,919 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,919 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:45,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:45,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58e82558[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:45,921 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:45,921 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,922 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,922 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:45,922 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,922 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:45,922 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:45,922 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:45,924 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,924 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:45,924 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:45,924 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:45,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:45,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:45,926 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:45,926 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:45,926 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:45,926 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:45,926 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:45,928 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,928 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:45,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:45,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,928 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:45,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:45,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:45,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:45,928 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:45,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,930 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,931 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/cf203d41b99b480e9af29dfa14c571a6 2023-04-18 22:55:45,933 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:45,933 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,933 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:45,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:45,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:45,933 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:45,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:45,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:45,933 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,933 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:45,933 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:45,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:45,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:45,933 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:45,934 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:45,934 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,934 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:45,934 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:45,935 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/8123afaaa41c41e9a110a27418c52559 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/8123afaaa41c41e9a110a27418c52559 2023-04-18 22:55:45,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/8123afaaa41c41e9a110a27418c52559, entries=200, sequenceid=293, filesize=14.3 K 2023-04-18 22:55:45,939 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/4f9fbe176217467b944e6730bf670a78 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/4f9fbe176217467b944e6730bf670a78 2023-04-18 22:55:45,942 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/4f9fbe176217467b944e6730bf670a78, entries=150, sequenceid=293, filesize=11.9 K 2023-04-18 22:55:45,943 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/cf203d41b99b480e9af29dfa14c571a6 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/cf203d41b99b480e9af29dfa14c571a6 2023-04-18 22:55:45,946 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/cf203d41b99b480e9af29dfa14c571a6, entries=150, sequenceid=293, filesize=11.9 K 2023-04-18 22:55:45,947 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 866ms, sequenceid=293, compaction requested=true 2023-04-18 22:55:45,947 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:45,947 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:45,947 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:45,947 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:45,947 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:45,947 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:45,948 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:45,948 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36868 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:45,948 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/A is initiating minor compaction (all files) 2023-04-18 22:55:45,948 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/B is initiating minor compaction (all files) 2023-04-18 22:55:45,948 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/A in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,948 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/B in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,948 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/0dc916ebc0264289b62f3cad08ccbb4d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/6a11d7f1c2c9463db78e6ab6a31c1ef0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/8123afaaa41c41e9a110a27418c52559] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=38.4 K 2023-04-18 22:55:45,948 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/1cab0c1cf48e46ada1c995d18294725f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/9c658c50c4784187b192e4ecbc0fd222, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/4f9fbe176217467b944e6730bf670a78] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=36.0 K 2023-04-18 22:55:45,949 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 0dc916ebc0264289b62f3cad08ccbb4d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681858543939 2023-04-18 22:55:45,949 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 1cab0c1cf48e46ada1c995d18294725f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681858543939 2023-04-18 22:55:45,949 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 9c658c50c4784187b192e4ecbc0fd222, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1681858544105 2023-04-18 22:55:45,949 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 6a11d7f1c2c9463db78e6ab6a31c1ef0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1681858544105 2023-04-18 22:55:45,949 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 4f9fbe176217467b944e6730bf670a78, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681858544762 2023-04-18 22:55:45,949 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 8123afaaa41c41e9a110a27418c52559, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681858544761 2023-04-18 22:55:45,954 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#B#compaction#419 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:45,959 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#A#compaction#420 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:45,970 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/cdc0669d3a164003a6b88fa51a11bcc3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/cdc0669d3a164003a6b88fa51a11bcc3 2023-04-18 22:55:45,971 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/8ce97dfa6ce54bd4818b5532c1822030 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/8ce97dfa6ce54bd4818b5532c1822030 2023-04-18 22:55:45,975 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/B of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into cdc0669d3a164003a6b88fa51a11bcc3(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:45,975 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:45,975 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/B, priority=13, startTime=1681858545947; duration=0sec 2023-04-18 22:55:45,975 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:45,975 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:45,976 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/A of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into 8ce97dfa6ce54bd4818b5532c1822030(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:45,976 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:45,976 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/A, priority=13, startTime=1681858545947; duration=0sec 2023-04-18 22:55:45,976 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:45,977 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36868 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:45,977 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/C is initiating minor compaction (all files) 2023-04-18 22:55:45,977 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/C in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:45,977 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/9e7a412eee164eeab85bafb5187ce5b6, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/53f9ab31274d402a93996c5cf7db1190, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/cf203d41b99b480e9af29dfa14c571a6] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=36.0 K 2023-04-18 22:55:45,977 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 9e7a412eee164eeab85bafb5187ce5b6, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681858543939 2023-04-18 22:55:45,978 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 53f9ab31274d402a93996c5cf7db1190, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1681858544105 2023-04-18 22:55:45,978 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting cf203d41b99b480e9af29dfa14c571a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681858544762 2023-04-18 22:55:45,985 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#C#compaction#421 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:45,995 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/f71cc36de0fd47589d9a46e7d516ea06 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/f71cc36de0fd47589d9a46e7d516ea06 2023-04-18 22:55:46,000 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/C of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into f71cc36de0fd47589d9a46e7d516ea06(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:46,000 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:46,000 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/C, priority=13, startTime=1681858545947; duration=0sec 2023-04-18 22:55:46,001 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:46,033 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:46,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:46,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:46,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:46,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:46,038 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:46,038 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:46,038 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:46,038 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:46,039 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:46,039 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:46,040 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:46,040 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:46,041 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,041 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:46,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:46,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:46,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:46,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:46,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:46,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:46,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:46,043 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:46,043 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:46,043 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:46,044 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:46,044 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:46,044 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:46,044 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:46,044 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 22:55:46,044 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:46,044 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:46,044 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:46,045 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:46,045 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:46,045 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:46,058 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/93a24a61b66541b49e232cb01fc144bb 2023-04-18 22:55:46,211 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-18 22:55:46,214 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/cac5eb5d37794b9fa5e65ad1466169b8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/9294f9e88d194d5eb56f57b6312dc48d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/41cafb0d9ccd481ab64d73947afe2288, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/bc5bb892c6564300bd65a9b84c5721ad, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f7fe8449e424433fb937697146eede4d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/2745eefd1bbe420ebb23bd13800f01e7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/e103d911f8b54a51b341926e9d1ec3ed, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/42a9a4c9d20f465785b00048ea87db77, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/a76db7b1705645e08e06e16ad44ff888, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f498d7b1615744ff817ba7d708082d8d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/b515a99a50a745ee97cee9169a89f3e1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3410d5d08378447096a3b790128fbc53, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/0dc916ebc0264289b62f3cad08ccbb4d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/6a11d7f1c2c9463db78e6ab6a31c1ef0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/8123afaaa41c41e9a110a27418c52559] to archive 2023-04-18 22:55:46,215 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:55:46,217 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/cac5eb5d37794b9fa5e65ad1466169b8 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/cac5eb5d37794b9fa5e65ad1466169b8 2023-04-18 22:55:46,218 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/9294f9e88d194d5eb56f57b6312dc48d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/9294f9e88d194d5eb56f57b6312dc48d 2023-04-18 22:55:46,219 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/41cafb0d9ccd481ab64d73947afe2288 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/41cafb0d9ccd481ab64d73947afe2288 2023-04-18 22:55:46,220 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/bc5bb892c6564300bd65a9b84c5721ad to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/bc5bb892c6564300bd65a9b84c5721ad 2023-04-18 22:55:46,221 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f7fe8449e424433fb937697146eede4d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f7fe8449e424433fb937697146eede4d 2023-04-18 22:55:46,222 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/2745eefd1bbe420ebb23bd13800f01e7 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/2745eefd1bbe420ebb23bd13800f01e7 2023-04-18 22:55:46,224 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/e103d911f8b54a51b341926e9d1ec3ed to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/e103d911f8b54a51b341926e9d1ec3ed 2023-04-18 22:55:46,225 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/42a9a4c9d20f465785b00048ea87db77 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/42a9a4c9d20f465785b00048ea87db77 2023-04-18 22:55:46,226 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/a76db7b1705645e08e06e16ad44ff888 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/a76db7b1705645e08e06e16ad44ff888 2023-04-18 22:55:46,227 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f498d7b1615744ff817ba7d708082d8d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f498d7b1615744ff817ba7d708082d8d 2023-04-18 22:55:46,228 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/b515a99a50a745ee97cee9169a89f3e1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/b515a99a50a745ee97cee9169a89f3e1 2023-04-18 22:55:46,229 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3410d5d08378447096a3b790128fbc53 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3410d5d08378447096a3b790128fbc53 2023-04-18 22:55:46,230 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/0dc916ebc0264289b62f3cad08ccbb4d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/0dc916ebc0264289b62f3cad08ccbb4d 2023-04-18 22:55:46,231 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/6a11d7f1c2c9463db78e6ab6a31c1ef0 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/6a11d7f1c2c9463db78e6ab6a31c1ef0 2023-04-18 22:55:46,232 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/8123afaaa41c41e9a110a27418c52559 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/8123afaaa41c41e9a110a27418c52559 2023-04-18 22:55:46,236 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ddf10b00c2854e6d99a7ac61f18737fc, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/4792272a9c3d44289da62fb2af3de3f3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/36b3e895a99f43938df748bfbbcfed10, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/626ff059f61d48018f21fd5a34bf902f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ee20a6d4d307491b88592f1603d9dacd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/837a0d21e295409cba6eb77345de55f3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ea373cf7dc3d4535a7007a0d37f7f27a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/9cffeb0751934ecfaff7f4f753ca6fa9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/6bb7f7beff1348fbaab679f3529f97a8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/924df5d0684d4d658bc3e1c645f3cc37, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/30baad6749cc43769c6756cbb7b518fd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/1cab0c1cf48e46ada1c995d18294725f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/68e0d02d45ba4d5889d97d5336fb1c17, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/9c658c50c4784187b192e4ecbc0fd222, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/4f9fbe176217467b944e6730bf670a78] to archive 2023-04-18 22:55:46,237 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:55:46,239 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ddf10b00c2854e6d99a7ac61f18737fc to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ddf10b00c2854e6d99a7ac61f18737fc 2023-04-18 22:55:46,241 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/4792272a9c3d44289da62fb2af3de3f3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/4792272a9c3d44289da62fb2af3de3f3 2023-04-18 22:55:46,242 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/36b3e895a99f43938df748bfbbcfed10 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/36b3e895a99f43938df748bfbbcfed10 2023-04-18 22:55:46,243 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/626ff059f61d48018f21fd5a34bf902f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/626ff059f61d48018f21fd5a34bf902f 2023-04-18 22:55:46,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:46,244 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:46,245 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ee20a6d4d307491b88592f1603d9dacd to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ee20a6d4d307491b88592f1603d9dacd 2023-04-18 22:55:46,246 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/837a0d21e295409cba6eb77345de55f3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/837a0d21e295409cba6eb77345de55f3 2023-04-18 22:55:46,247 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ea373cf7dc3d4535a7007a0d37f7f27a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/ea373cf7dc3d4535a7007a0d37f7f27a 2023-04-18 22:55:46,248 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/9cffeb0751934ecfaff7f4f753ca6fa9 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/9cffeb0751934ecfaff7f4f753ca6fa9 2023-04-18 22:55:46,249 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/6bb7f7beff1348fbaab679f3529f97a8 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/6bb7f7beff1348fbaab679f3529f97a8 2023-04-18 22:55:46,250 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/924df5d0684d4d658bc3e1c645f3cc37 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/924df5d0684d4d658bc3e1c645f3cc37 2023-04-18 22:55:46,252 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/30baad6749cc43769c6756cbb7b518fd to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/30baad6749cc43769c6756cbb7b518fd 2023-04-18 22:55:46,253 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/1cab0c1cf48e46ada1c995d18294725f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/1cab0c1cf48e46ada1c995d18294725f 2023-04-18 22:55:46,254 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/68e0d02d45ba4d5889d97d5336fb1c17 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/68e0d02d45ba4d5889d97d5336fb1c17 2023-04-18 22:55:46,256 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/9c658c50c4784187b192e4ecbc0fd222 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/9c658c50c4784187b192e4ecbc0fd222 2023-04-18 22:55:46,258 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/4f9fbe176217467b944e6730bf670a78 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/4f9fbe176217467b944e6730bf670a78 2023-04-18 22:55:46,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:46,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858606255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:46,263 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/02c94e12fdcf4d418c13ef2e2736aaa3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/77bdf6c1ad4e4d49a0b6c74c2ddbd5a0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/b96de023d15b4e0d81b041da32744ff7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/c21483fc8a1e4b87b425cea08dcc29f9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ee766c6a09ba4cc7817bc49f236a8784, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/11b98bc0d9d44dfd83360a22ca3bff56, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ce11bedbcede4a429dd6dd873a48dddd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/12ffcf80ada64e2386ad9c020a94738a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/608d85f0dfe14794a3c1c64197297827, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/fd089b48bdf049a491d6a3da1c6076a7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/8376abd9522f4a8d95bc9d35954c55d8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/9e7a412eee164eeab85bafb5187ce5b6, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/008a4afef0db4c39af0c98c583016e3f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/53f9ab31274d402a93996c5cf7db1190, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/cf203d41b99b480e9af29dfa14c571a6] to archive 2023-04-18 22:55:46,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858606257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:46,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858606258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,264 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:55:46,266 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/02c94e12fdcf4d418c13ef2e2736aaa3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/02c94e12fdcf4d418c13ef2e2736aaa3 2023-04-18 22:55:46,267 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/77bdf6c1ad4e4d49a0b6c74c2ddbd5a0 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/77bdf6c1ad4e4d49a0b6c74c2ddbd5a0 2023-04-18 22:55:46,268 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/b96de023d15b4e0d81b041da32744ff7 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/b96de023d15b4e0d81b041da32744ff7 2023-04-18 22:55:46,269 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/c21483fc8a1e4b87b425cea08dcc29f9 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/c21483fc8a1e4b87b425cea08dcc29f9 2023-04-18 22:55:46,270 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ee766c6a09ba4cc7817bc49f236a8784 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ee766c6a09ba4cc7817bc49f236a8784 2023-04-18 22:55:46,271 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/11b98bc0d9d44dfd83360a22ca3bff56 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/11b98bc0d9d44dfd83360a22ca3bff56 2023-04-18 22:55:46,272 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ce11bedbcede4a429dd6dd873a48dddd to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ce11bedbcede4a429dd6dd873a48dddd 2023-04-18 22:55:46,273 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/12ffcf80ada64e2386ad9c020a94738a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/12ffcf80ada64e2386ad9c020a94738a 2023-04-18 22:55:46,274 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/608d85f0dfe14794a3c1c64197297827 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/608d85f0dfe14794a3c1c64197297827 2023-04-18 22:55:46,275 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/fd089b48bdf049a491d6a3da1c6076a7 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/fd089b48bdf049a491d6a3da1c6076a7 2023-04-18 22:55:46,278 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/8376abd9522f4a8d95bc9d35954c55d8 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/8376abd9522f4a8d95bc9d35954c55d8 2023-04-18 22:55:46,279 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/9e7a412eee164eeab85bafb5187ce5b6 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/9e7a412eee164eeab85bafb5187ce5b6 2023-04-18 22:55:46,280 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/008a4afef0db4c39af0c98c583016e3f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/008a4afef0db4c39af0c98c583016e3f 2023-04-18 22:55:46,281 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/53f9ab31274d402a93996c5cf7db1190 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/53f9ab31274d402a93996c5cf7db1190 2023-04-18 22:55:46,282 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/cf203d41b99b480e9af29dfa14c571a6 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/cf203d41b99b480e9af29dfa14c571a6 2023-04-18 22:55:46,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:46,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858606360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:46,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858606365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:46,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858606365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,476 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/a90d56f5256848d8b538a4be83185666 2023-04-18 22:55:46,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:46,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858606566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:46,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858606569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:46,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858606570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:46,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858606871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:46,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858606873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:46,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858606873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,887 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/ccae41e792bd47329f096755ae4942d2 2023-04-18 22:55:46,931 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/93a24a61b66541b49e232cb01fc144bb as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/93a24a61b66541b49e232cb01fc144bb 2023-04-18 22:55:46,937 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/93a24a61b66541b49e232cb01fc144bb, entries=150, sequenceid=315, filesize=11.9 K 2023-04-18 22:55:46,939 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/a90d56f5256848d8b538a4be83185666 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/a90d56f5256848d8b538a4be83185666 2023-04-18 22:55:46,943 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/a90d56f5256848d8b538a4be83185666, entries=150, sequenceid=315, filesize=11.9 K 2023-04-18 22:55:46,944 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/ccae41e792bd47329f096755ae4942d2 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ccae41e792bd47329f096755ae4942d2 2023-04-18 22:55:46,949 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ccae41e792bd47329f096755ae4942d2, entries=150, sequenceid=315, filesize=11.9 K 2023-04-18 22:55:46,950 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 905ms, sequenceid=315, compaction requested=false 2023-04-18 22:55:46,950 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:46,950 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:46,950 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:46,950 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:46,950 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,950 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:46,950 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:46,953 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,953 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:46,953 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:46,953 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:46,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:46,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:46,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70d06309[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:46,955 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:46,955 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:46,957 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:46,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:46,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:46,957 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:46,957 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:46,957 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:46,957 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,957 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:46,960 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,960 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:46,960 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:46,960 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:46,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:46,963 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:46,963 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:46,963 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:46,963 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:46,963 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:46,965 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:46,965 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:46,965 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:46,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:46,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:46,966 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,967 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,971 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:46,971 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:46,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:46,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:46,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:46,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:46,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:46,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:46,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:46,972 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:46,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:46,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:46,972 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:46,972 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:46,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:47,072 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:47,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:47,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:47,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:47,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:47,075 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:47,075 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:47,075 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:47,075 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:47,076 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:47,076 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:47,077 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:47,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:47,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:47,078 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:47,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:47,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:47,078 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,078 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:47,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:47,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:47,078 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:47,078 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:47,078 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:47,080 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:47,080 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:47,080 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:47,080 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:47,080 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 22:55:47,080 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:47,080 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:47,080 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:47,080 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:47,080 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:47,080 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:47,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:47,380 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:47,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:47,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858607410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:47,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858607411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:47,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858607411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,494 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/701ea3b553ed4fe1ae461b1eb1fe81ef 2023-04-18 22:55:47,512 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/420d729a067d4811a47edd2279a12fc7 2023-04-18 22:55:47,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:47,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858607516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:47,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858607516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:47,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858607520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,529 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/a44bde606d2245728f37834be2e9ba68 2023-04-18 22:55:47,535 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/701ea3b553ed4fe1ae461b1eb1fe81ef as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/701ea3b553ed4fe1ae461b1eb1fe81ef 2023-04-18 22:55:47,539 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/701ea3b553ed4fe1ae461b1eb1fe81ef, entries=150, sequenceid=328, filesize=11.9 K 2023-04-18 22:55:47,540 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/420d729a067d4811a47edd2279a12fc7 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/420d729a067d4811a47edd2279a12fc7 2023-04-18 22:55:47,543 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/420d729a067d4811a47edd2279a12fc7, entries=150, sequenceid=328, filesize=11.9 K 2023-04-18 22:55:47,544 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/a44bde606d2245728f37834be2e9ba68 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/a44bde606d2245728f37834be2e9ba68 2023-04-18 22:55:47,550 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/a44bde606d2245728f37834be2e9ba68, entries=150, sequenceid=328, filesize=11.9 K 2023-04-18 22:55:47,551 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 471ms, sequenceid=328, compaction requested=true 2023-04-18 22:55:47,551 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:47,551 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:47,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:47,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:47,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:47,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:47,552 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,553 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:47,553 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:47,553 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:47,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:47,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:47,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10f92a5[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:47,555 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:47,555 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:47,560 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:47,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:47,567 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,567 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:47,567 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:47,567 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:47,567 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:47,567 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:47,567 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:47,567 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:47,567 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:47,567 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:47,567 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:47,570 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:47,570 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:47,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:47,570 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:47,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:47,571 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:47,572 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:47,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:47,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:47,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:47,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:47,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,576 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:47,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:47,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:47,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:47,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:47,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:47,576 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:47,576 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:47,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:47,676 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:47,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:47,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:47,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:47,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:47,680 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:47,680 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:47,680 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:47,680 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:47,680 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:47,680 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:47,682 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:47,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:47,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:47,682 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:47,683 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,683 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:47,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:47,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:47,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:47,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:47,684 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:47,684 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:47,684 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:47,685 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:47,685 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:47,685 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:47,685 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:47,685 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 22:55:47,685 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:47,685 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:47,685 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:47,685 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:47,686 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:47,686 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:47,714 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/aa2b74abfc5a4d919282ab1c8cba45e0 2023-04-18 22:55:47,726 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/524b3447f7724b88b4b15995c8fc7e52 2023-04-18 22:55:47,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:47,728 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:47,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:47,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858607739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:47,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858607740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:47,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858607749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:47,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858607850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:47,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858607850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:47,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:47,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858607858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:48,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858608059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:48,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858608059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:48,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858608073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,146 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/e9f3e612e283448a8b75f811e0c04cdb 2023-04-18 22:55:48,150 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/aa2b74abfc5a4d919282ab1c8cba45e0 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/aa2b74abfc5a4d919282ab1c8cba45e0 2023-04-18 22:55:48,153 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/aa2b74abfc5a4d919282ab1c8cba45e0, entries=150, sequenceid=347, filesize=11.9 K 2023-04-18 22:55:48,154 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/524b3447f7724b88b4b15995c8fc7e52 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/524b3447f7724b88b4b15995c8fc7e52 2023-04-18 22:55:48,157 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/524b3447f7724b88b4b15995c8fc7e52, entries=150, sequenceid=347, filesize=11.9 K 2023-04-18 22:55:48,158 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/e9f3e612e283448a8b75f811e0c04cdb as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/e9f3e612e283448a8b75f811e0c04cdb 2023-04-18 22:55:48,161 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/e9f3e612e283448a8b75f811e0c04cdb, entries=150, sequenceid=347, filesize=11.9 K 2023-04-18 22:55:48,162 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 477ms, sequenceid=347, compaction requested=true 2023-04-18 22:55:48,162 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:48,162 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:48,162 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:48,162 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:48,162 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,162 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:48,162 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:48,163 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:48,164 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,164 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:48,164 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c12ab12[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:48,166 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:48,166 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:48,167 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:48,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:48,167 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,167 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:48,168 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,168 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:48,168 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:48,168 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:48,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:48,171 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:48,171 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:48,171 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:48,171 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:48,171 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:48,174 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:48,174 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:48,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:48,174 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:48,174 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:48,175 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,177 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,178 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:48,178 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:48,178 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,178 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:48,178 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:48,178 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:48,178 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:48,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:48,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:48,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:48,178 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:48,178 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:48,178 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:48,178 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:48,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:48,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:48,278 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:48,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:48,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:48,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:48,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:48,282 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:48,282 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:48,282 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:48,282 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:48,282 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:48,282 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:48,284 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:48,284 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:48,284 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,284 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:48,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:48,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:48,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:48,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:48,285 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:48,285 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:48,285 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:48,285 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:48,285 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:48,285 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:48,285 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:48,286 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-18 22:55:48,286 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:48,286 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:48,286 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:48,286 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:48,286 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:48,286 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:48,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:48,366 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:48,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:48,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858608409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:48,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858608415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:48,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858608415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:48,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858608521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:48,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858608522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:48,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858608529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,698 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/93a64148ceb74c11ba907545f3832b3c 2023-04-18 22:55:48,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:48,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858608731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:48,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858608733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:48,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:48,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858608738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:49,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858609042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:49,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858609043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:49,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858609045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,118 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/1135db78873b447e825bdc0e50dccb58 2023-04-18 22:55:49,149 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/b2b5fa0a20634d4198d9a90c64480a8c 2023-04-18 22:55:49,155 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/93a64148ceb74c11ba907545f3832b3c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/93a64148ceb74c11ba907545f3832b3c 2023-04-18 22:55:49,160 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/93a64148ceb74c11ba907545f3832b3c, entries=150, sequenceid=360, filesize=11.9 K 2023-04-18 22:55:49,161 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/1135db78873b447e825bdc0e50dccb58 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/1135db78873b447e825bdc0e50dccb58 2023-04-18 22:55:49,165 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/1135db78873b447e825bdc0e50dccb58, entries=150, sequenceid=360, filesize=11.9 K 2023-04-18 22:55:49,166 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/b2b5fa0a20634d4198d9a90c64480a8c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/b2b5fa0a20634d4198d9a90c64480a8c 2023-04-18 22:55:49,170 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/b2b5fa0a20634d4198d9a90c64480a8c, entries=150, sequenceid=360, filesize=11.9 K 2023-04-18 22:55:49,171 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 885ms, sequenceid=360, compaction requested=true 2023-04-18 22:55:49,171 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:49,171 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:49,171 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:49,171 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:49,171 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,171 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:49,171 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:49,173 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,173 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,174 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,174 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:49,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15c98d2d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:49,175 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:49,175 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,181 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,181 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:49,181 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:49,181 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:49,181 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,181 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:49,183 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:49,183 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,183 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,183 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:49,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:49,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:49,185 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:49,186 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:49,186 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:49,186 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:49,186 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:49,187 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,187 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,187 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,187 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,188 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,190 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,190 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,190 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,190 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:49,191 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:49,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:49,191 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:49,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:49,191 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,191 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,191 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,191 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:49,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:49,191 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:49,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:49,191 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:49,291 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:49,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:49,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:49,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:49,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:49,294 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:49,295 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:49,295 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:49,295 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:49,295 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,295 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,301 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:49,301 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:49,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:49,301 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,301 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:49,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:49,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:49,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:49,302 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:49,302 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:49,302 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:49,302 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:49,302 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:49,302 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:49,302 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:49,302 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-18 22:55:49,302 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:49,303 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:49,303 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:49,303 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:49,303 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:49,303 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:49,321 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/88bea1d0960d4298874becad6efacbba 2023-04-18 22:55:49,333 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/a79eaa69d78b4bc4ba9428b2712c0edc 2023-04-18 22:55:49,345 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/84bf13f3eae24ccda374ce4fb529af81 2023-04-18 22:55:49,349 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/88bea1d0960d4298874becad6efacbba as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/88bea1d0960d4298874becad6efacbba 2023-04-18 22:55:49,355 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/88bea1d0960d4298874becad6efacbba, entries=150, sequenceid=380, filesize=11.9 K 2023-04-18 22:55:49,355 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/a79eaa69d78b4bc4ba9428b2712c0edc as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/a79eaa69d78b4bc4ba9428b2712c0edc 2023-04-18 22:55:49,359 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/a79eaa69d78b4bc4ba9428b2712c0edc, entries=150, sequenceid=380, filesize=11.9 K 2023-04-18 22:55:49,360 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/84bf13f3eae24ccda374ce4fb529af81 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/84bf13f3eae24ccda374ce4fb529af81 2023-04-18 22:55:49,363 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/84bf13f3eae24ccda374ce4fb529af81, entries=150, sequenceid=380, filesize=11.9 K 2023-04-18 22:55:49,364 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 62ms, sequenceid=380, compaction requested=true 2023-04-18 22:55:49,364 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:49,364 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:49,364 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:49,364 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:49,364 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,364 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:49,364 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:49,366 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,367 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,367 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,367 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:49,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66726cb6[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:49,368 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:49,368 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,369 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,369 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:49,369 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:49,369 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,369 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:49,369 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:49,371 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:49,371 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,371 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:49,371 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:49,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:49,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:49,374 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:49,374 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:49,374 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:49,374 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:49,374 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:49,382 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,382 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,382 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,383 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,384 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,385 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,385 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,385 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,385 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,385 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,385 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,385 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:49,386 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:49,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:49,386 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:49,386 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:49,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:49,387 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:49,387 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:49,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:49,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:49,487 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:49,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:49,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:49,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:49,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:49,493 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:49,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:49,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:49,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:49,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,494 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:49,495 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:49,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,496 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,496 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:49,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:49,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:49,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:49,497 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:49,497 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:49,497 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:49,498 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:49,498 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:49,498 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:49,498 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:49,498 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:49,498 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:49,498 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:49,498 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:49,498 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,498 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:49,498 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:49,500 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,500 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,501 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,501 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:49,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:49,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b154067[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:49,505 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:49,505 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,506 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,506 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:49,506 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:49,506 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:49,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:49,508 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,508 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:49,508 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:49,508 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:49,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:49,512 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:49,512 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:49,512 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:49,512 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:49,512 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:49,513 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,513 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:49,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,515 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,515 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,517 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,520 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:49,520 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:49,520 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:49,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:49,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:49,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:49,520 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,521 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:49,521 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:49,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:49,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:49,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:49,566 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:55:49,566 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:49,566 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:49,566 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:49,566 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:49,566 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:49,566 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:49,593 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/0f64503b00a4427297f6690ea9436673 2023-04-18 22:55:49,621 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:49,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:49,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:49,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:49,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:49,624 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:49,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:49,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:49,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:49,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,626 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:49,626 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:49,627 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,627 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:49,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:49,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:49,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:49,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:49,628 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:49,628 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:49,628 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:49,628 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:49,629 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:49,631 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,631 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858549612, max timestamp=1681858549631Num uniques 859; 2023-04-18 22:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858549612, max timestamp=1681858549631Num uniques 859; 2023-04-18 22:55:49,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:55:49,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858549612, max timestamp=1681858549631Num uniques 859; 2023-04-18 22:55:49,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:49,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c6222c[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:49,635 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:49,635 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,637 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,637 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:49,637 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:49,637 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:49,637 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,637 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:49,638 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:49,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:49,641 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:49,641 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:49,641 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:49,641 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:49,641 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:49,644 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:49,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,653 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,657 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,657 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:49,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:49,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:49,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:49,657 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:49,657 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:49,658 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:49,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:49,657 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:49,658 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,658 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:49,658 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:49,658 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,658 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,658 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:49,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858609706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:49,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858609708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:49,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858609709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,758 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:49,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:49,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:49,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:49,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:49,764 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:49,764 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:49,764 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:49,764 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:49,765 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,765 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,767 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,767 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:49,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:49,767 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,767 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:49,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:49,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:49,772 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:49,772 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:49,772 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:49,784 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:49,787 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:49,787 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:49,787 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:49,787 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:49,787 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:49,787 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:49,787 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,787 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:49,787 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:49,789 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,789 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,789 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,789 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:49,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b3f67bc[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:49,791 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:49,791 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,793 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,793 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:49,793 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:49,793 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:49,793 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,793 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:49,797 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:49,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:49,800 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:49,800 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:49,800 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:49,800 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:49,800 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:49,801 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:49,801 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:49,801 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:49,802 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,802 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,802 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:49,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:49,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:49,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,805 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,811 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:49,811 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,811 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:49,811 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:49,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:49,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:49,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:49,811 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,811 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,811 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,811 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,811 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,811 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,811 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:49,811 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:49,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:49,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:49,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858609826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:49,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858609826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:49,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858609826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,911 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:49,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:49,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:49,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:49,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:49,924 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:49,924 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:49,924 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:49,925 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:49,925 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,925 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,927 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,927 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:49,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:49,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:49,927 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,927 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:49,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:49,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:49,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:49,946 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:49,946 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:49,946 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:49,946 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:49,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:49,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:49,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:49,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:49,948 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,948 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,949 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,949 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:49,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2edda7b1[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:49,952 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:49,952 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,953 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,953 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:49,953 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:49,953 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:49,954 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,954 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:49,954 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:49,954 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:49,954 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:49,954 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:49,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:49,957 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:49,957 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:49,957 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:49,957 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:49,957 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:49,962 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,962 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:49,962 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:49,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,963 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,965 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,967 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:49,968 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:49,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:49,968 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:49,968 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:49,968 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:49,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:49,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:49,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:49,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:49,968 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:49,968 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:49,968 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,968 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:49,968 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:49,968 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:49,969 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:50,004 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/400e68d7260f4ff9b5c44e556501fb20 2023-04-18 22:55:50,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:50,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858610039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,050 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/d4d9bf9c066f4ae4b18034c1d82758c3 2023-04-18 22:55:50,053 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/0f64503b00a4427297f6690ea9436673 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/0f64503b00a4427297f6690ea9436673 2023-04-18 22:55:50,056 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/0f64503b00a4427297f6690ea9436673, entries=150, sequenceid=391, filesize=11.9 K 2023-04-18 22:55:50,057 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/400e68d7260f4ff9b5c44e556501fb20 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/400e68d7260f4ff9b5c44e556501fb20 2023-04-18 22:55:50,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:50,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858610040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:50,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858610040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/400e68d7260f4ff9b5c44e556501fb20, entries=100, sequenceid=391, filesize=9.5 K 2023-04-18 22:55:50,063 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/d4d9bf9c066f4ae4b18034c1d82758c3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/d4d9bf9c066f4ae4b18034c1d82758c3 2023-04-18 22:55:50,067 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/d4d9bf9c066f4ae4b18034c1d82758c3, entries=100, sequenceid=391, filesize=9.5 K 2023-04-18 22:55:50,068 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 503ms, sequenceid=391, compaction requested=true 2023-04-18 22:55:50,068 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:50,068 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:50,068 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-18 22:55:50,068 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-18 22:55:50,068 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:50,068 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:50,069 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:50,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:50,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:50,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:50,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:50,085 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:50,085 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:50,085 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:50,085 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:50,085 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,085 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,087 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,087 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:50,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:50,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:50,087 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,087 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:50,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,090 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85810 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-18 22:55:50,090 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/A is initiating minor compaction (all files) 2023-04-18 22:55:50,090 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/A in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:50,090 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/8ce97dfa6ce54bd4818b5532c1822030, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/93a24a61b66541b49e232cb01fc144bb, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/701ea3b553ed4fe1ae461b1eb1fe81ef, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/aa2b74abfc5a4d919282ab1c8cba45e0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/93a64148ceb74c11ba907545f3832b3c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/88bea1d0960d4298874becad6efacbba, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/0f64503b00a4427297f6690ea9436673] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=83.8 K 2023-04-18 22:55:50,090 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83370 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-18 22:55:50,090 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/B is initiating minor compaction (all files) 2023-04-18 22:55:50,091 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 8ce97dfa6ce54bd4818b5532c1822030, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681858544762 2023-04-18 22:55:50,091 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/B in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:50,091 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/cdc0669d3a164003a6b88fa51a11bcc3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/a90d56f5256848d8b538a4be83185666, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/420d729a067d4811a47edd2279a12fc7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/524b3447f7724b88b4b15995c8fc7e52, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/1135db78873b447e825bdc0e50dccb58, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/a79eaa69d78b4bc4ba9428b2712c0edc, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/400e68d7260f4ff9b5c44e556501fb20] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=81.4 K 2023-04-18 22:55:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:50,091 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 93a24a61b66541b49e232cb01fc144bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1681858545103 2023-04-18 22:55:50,091 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting cdc0669d3a164003a6b88fa51a11bcc3, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681858544762 2023-04-18 22:55:50,092 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 701ea3b553ed4fe1ae461b1eb1fe81ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1681858546254 2023-04-18 22:55:50,092 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting a90d56f5256848d8b538a4be83185666, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1681858545103 2023-04-18 22:55:50,092 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting aa2b74abfc5a4d919282ab1c8cba45e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1681858547397 2023-04-18 22:55:50,092 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 420d729a067d4811a47edd2279a12fc7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1681858546254 2023-04-18 22:55:50,092 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 93a64148ceb74c11ba907545f3832b3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1681858547737 2023-04-18 22:55:50,093 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 524b3447f7724b88b4b15995c8fc7e52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1681858547397 2023-04-18 22:55:50,093 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 88bea1d0960d4298874becad6efacbba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681858548408 2023-04-18 22:55:50,093 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 1135db78873b447e825bdc0e50dccb58, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1681858547737 2023-04-18 22:55:50,093 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 0f64503b00a4427297f6690ea9436673, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681858549557 2023-04-18 22:55:50,093 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting a79eaa69d78b4bc4ba9428b2712c0edc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681858548408 2023-04-18 22:55:50,094 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 400e68d7260f4ff9b5c44e556501fb20, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681858549557 2023-04-18 22:55:50,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:50,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:50,104 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:50,104 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:50,104 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:50,113 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#A#compaction#440 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:50,126 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#B#compaction#441 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:50,136 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:50,136 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:50,136 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:50,136 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:50,136 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-18 22:55:50,136 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:50,136 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:50,136 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:50,136 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:50,136 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:50,136 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:50,170 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/4e225cddca98488f991c2568c9346ad3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/4e225cddca98488f991c2568c9346ad3 2023-04-18 22:55:50,176 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/B of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into 4e225cddca98488f991c2568c9346ad3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:50,176 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:50,176 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/B, priority=9, startTime=1681858550068; duration=0sec 2023-04-18 22:55:50,176 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:50,176 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-18 22:55:50,177 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/4188da27a08348fcaee36508088a40c5 2023-04-18 22:55:50,179 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83370 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-18 22:55:50,179 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/C is initiating minor compaction (all files) 2023-04-18 22:55:50,179 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/C in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:50,179 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/f71cc36de0fd47589d9a46e7d516ea06, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ccae41e792bd47329f096755ae4942d2, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/a44bde606d2245728f37834be2e9ba68, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/e9f3e612e283448a8b75f811e0c04cdb, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/b2b5fa0a20634d4198d9a90c64480a8c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/84bf13f3eae24ccda374ce4fb529af81, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/d4d9bf9c066f4ae4b18034c1d82758c3] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=81.4 K 2023-04-18 22:55:50,180 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting f71cc36de0fd47589d9a46e7d516ea06, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681858544762 2023-04-18 22:55:50,181 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting ccae41e792bd47329f096755ae4942d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1681858545103 2023-04-18 22:55:50,181 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting a44bde606d2245728f37834be2e9ba68, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1681858546254 2023-04-18 22:55:50,182 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting e9f3e612e283448a8b75f811e0c04cdb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1681858547397 2023-04-18 22:55:50,182 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting b2b5fa0a20634d4198d9a90c64480a8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1681858547737 2023-04-18 22:55:50,182 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 84bf13f3eae24ccda374ce4fb529af81, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681858548408 2023-04-18 22:55:50,183 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting d4d9bf9c066f4ae4b18034c1d82758c3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681858549557 2023-04-18 22:55:50,206 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#C#compaction#444 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:50,240 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/8b47f86f8d614389ba89f4b9c7b967db 2023-04-18 22:55:50,252 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/c9ada5d185f44da088c6e9f9cbd8a163 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/c9ada5d185f44da088c6e9f9cbd8a163 2023-04-18 22:55:50,263 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/C of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into c9ada5d185f44da088c6e9f9cbd8a163(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:50,263 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:50,263 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/C, priority=9, startTime=1681858550068; duration=0sec 2023-04-18 22:55:50,264 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:50,319 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/540828737208482d9582b9f440c960b3 2023-04-18 22:55:50,325 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/4188da27a08348fcaee36508088a40c5 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/4188da27a08348fcaee36508088a40c5 2023-04-18 22:55:50,331 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/4188da27a08348fcaee36508088a40c5, entries=150, sequenceid=427, filesize=11.9 K 2023-04-18 22:55:50,332 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/8b47f86f8d614389ba89f4b9c7b967db as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/8b47f86f8d614389ba89f4b9c7b967db 2023-04-18 22:55:50,337 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/8b47f86f8d614389ba89f4b9c7b967db, entries=150, sequenceid=427, filesize=11.9 K 2023-04-18 22:55:50,339 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/540828737208482d9582b9f440c960b3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/540828737208482d9582b9f440c960b3 2023-04-18 22:55:50,343 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/540828737208482d9582b9f440c960b3, entries=150, sequenceid=427, filesize=11.9 K 2023-04-18 22:55:50,344 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 208ms, sequenceid=427, compaction requested=false 2023-04-18 22:55:50,344 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:50,344 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:50,344 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:50,344 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:50,344 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,344 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:50,344 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:50,348 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:50,348 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:50,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:50,350 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,350 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:50,350 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:50,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ffadc75[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:50,351 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,356 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,356 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:50,356 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:50,356 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:50,357 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,357 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:50,358 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,358 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:50,360 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:50,360 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:50,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:50,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:50,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:50,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:50,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:50,365 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:50,365 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:50,365 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:50,365 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:50,365 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:50,367 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,367 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,367 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:50,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:50,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:50,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,369 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:50,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:50,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:50,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,370 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:50,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,374 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:50,374 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:50,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:50,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:50,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:50,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:50,374 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,374 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,375 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,375 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:50,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:50,375 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:50,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:50,375 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,375 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,375 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:50,375 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:50,375 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:50,379 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:55:50,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:50,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:50,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:50,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:50,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:50,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:50,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:50,419 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/05b6a1c18d9a499293acee7fba3a3e1f 2023-04-18 22:55:50,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858550399, max timestamp=1681858550443Num uniques 859; 2023-04-18 22:55:50,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:55:50,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858550399, max timestamp=1681858550443Num uniques 859; 2023-04-18 22:55:50,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:55:50,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858550399, max timestamp=1681858550443Num uniques 859; 2023-04-18 22:55:50,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:55:50,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/cc2be541610e428f8d8447ac64413c15 2023-04-18 22:55:50,475 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:50,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:50,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:50,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:50,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:50,485 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:50,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:50,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:50,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:50,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,488 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:50,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:50,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,490 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,491 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:50,491 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:50,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:50,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:50,512 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:50,512 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:50,513 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/e5dfc00a2dbe40d5b36bcd25a3ec2304 2023-04-18 22:55:50,517 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/05b6a1c18d9a499293acee7fba3a3e1f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/05b6a1c18d9a499293acee7fba3a3e1f 2023-04-18 22:55:50,519 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:50,520 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:50,520 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:50,520 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:50,520 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:50,520 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:50,520 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,520 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:50,520 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:50,520 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/05b6a1c18d9a499293acee7fba3a3e1f, entries=200, sequenceid=439, filesize=14.3 K 2023-04-18 22:55:50,521 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/cc2be541610e428f8d8447ac64413c15 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/cc2be541610e428f8d8447ac64413c15 2023-04-18 22:55:50,522 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,522 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,522 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,522 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:50,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:50,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c1e5ae4[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:50,525 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:50,525 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,526 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,526 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:50,526 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:50,526 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,526 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:50,526 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:50,527 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,527 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:50,527 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:50,527 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:50,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:50,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:50,530 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:50,530 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:50,530 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:50,531 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:50,531 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:50,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:50,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858610507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:50,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858610508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,533 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,533 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:50,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,534 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,535 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,536 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:50,541 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:50,541 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:50,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:50,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:50,542 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:50,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:50,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:50,542 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:50,542 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:50,542 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,543 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,543 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,543 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,543 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,543 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,543 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:50,548 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/cc2be541610e428f8d8447ac64413c15, entries=150, sequenceid=439, filesize=11.9 K 2023-04-18 22:55:50,549 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/e5dfc00a2dbe40d5b36bcd25a3ec2304 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/e5dfc00a2dbe40d5b36bcd25a3ec2304 2023-04-18 22:55:50,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:50,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858610534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,554 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/e5dfc00a2dbe40d5b36bcd25a3ec2304, entries=150, sequenceid=439, filesize=11.9 K 2023-04-18 22:55:50,555 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 176ms, sequenceid=439, compaction requested=true 2023-04-18 22:55:50,555 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:50,555 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:50,555 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-18 22:55:50,555 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-18 22:55:50,555 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 7 compacting, 2 eligible, 16 blocking 2023-04-18 22:55:50,556 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-18 22:55:50,556 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-18 22:55:50,556 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. because compaction request was cancelled 2023-04-18 22:55:50,556 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:50,557 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36798 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:50,557 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/C is initiating minor compaction (all files) 2023-04-18 22:55:50,558 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/C in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:50,558 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/c9ada5d185f44da088c6e9f9cbd8a163, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/540828737208482d9582b9f440c960b3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/e5dfc00a2dbe40d5b36bcd25a3ec2304] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=35.9 K 2023-04-18 22:55:50,558 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting c9ada5d185f44da088c6e9f9cbd8a163, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681858548413 2023-04-18 22:55:50,558 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 540828737208482d9582b9f440c960b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681858549683 2023-04-18 22:55:50,559 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting e5dfc00a2dbe40d5b36bcd25a3ec2304, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1681858550366 2023-04-18 22:55:50,585 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/0c6ac61f183b41fbb7df8298f74cd6a9 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/0c6ac61f183b41fbb7df8298f74cd6a9 2023-04-18 22:55:50,591 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/A of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into 0c6ac61f183b41fbb7df8298f74cd6a9(size=12.1 K), total size for store is 38.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:50,591 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:50,591 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/A, priority=9, startTime=1681858550068; duration=0sec 2023-04-18 22:55:50,591 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:50,591 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:50,592 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36798 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:50,592 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/B is initiating minor compaction (all files) 2023-04-18 22:55:50,592 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/B in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:50,592 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/4e225cddca98488f991c2568c9346ad3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/8b47f86f8d614389ba89f4b9c7b967db, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/cc2be541610e428f8d8447ac64413c15] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=35.9 K 2023-04-18 22:55:50,593 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 4e225cddca98488f991c2568c9346ad3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681858548413 2023-04-18 22:55:50,593 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 8b47f86f8d614389ba89f4b9c7b967db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681858549683 2023-04-18 22:55:50,594 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting cc2be541610e428f8d8447ac64413c15, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1681858550366 2023-04-18 22:55:50,601 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#C#compaction#449 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:50,609 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#B#compaction#450 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:50,642 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:50,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:50,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:50,649 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-18 22:55:50,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:50,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:50,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:50,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:50,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:50,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:50,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:50,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:50,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858610672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:50,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858610688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:50,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858610689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:50,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:50,739 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:50,739 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:50,739 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:50,739 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:50,740 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,740 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,740 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/3358aee613dd4bafa88201be8b0c4df1 2023-04-18 22:55:50,742 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:50,742 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:50,742 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,742 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:50,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:50,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:50,746 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:50,746 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:50,746 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:50,748 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:50,749 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:50,750 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:50,750 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:50,750 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:50,750 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:50,750 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:50,750 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,750 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:50,750 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:50,753 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,753 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:50,753 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,754 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:50,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cbcca5c[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:50,756 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:50,756 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,757 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:50,757 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:50,757 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,757 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:50,759 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:50,759 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:50,759 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:50,759 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:50,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:50,761 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:50,761 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:50,761 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:50,761 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:50,761 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:50,763 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,763 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:50,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,764 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:50,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:50,765 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,767 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,770 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:50,770 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:50,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:50,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:50,771 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,771 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,771 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:50,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:50,771 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:50,771 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:50,771 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:50,771 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,771 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,771 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,771 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:50,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858610790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:50,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858610811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:50,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858610811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,871 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:50,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:50,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:50,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:50,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:50,878 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:50,878 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:50,878 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:50,878 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:50,878 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,878 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,880 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:50,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:50,880 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,880 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,880 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:50,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:50,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:50,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:50,890 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:50,892 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:50,892 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:50,896 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:50,896 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:50,896 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:50,896 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:50,898 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,898 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,899 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:50,899 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:50,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:50,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f1f0455[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:50,904 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:50,904 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,905 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,905 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:50,906 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:50,906 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:50,906 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,906 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:50,913 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:50,915 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:50,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:50,918 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:50,919 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:50,919 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:50,919 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:50,919 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:50,921 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,921 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:50,921 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:50,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:50,922 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,923 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,929 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:50,929 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:50,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:50,929 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,929 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,929 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:50,929 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:50,929 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,929 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:50,929 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:50,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:50,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:50,929 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:50,930 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:50,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:50,929 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:51,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:51,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858611003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,030 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:51,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:51,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:51,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:51,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858611023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:51,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858611023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:51,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:51,037 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:51,037 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:51,037 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:51,037 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:51,037 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,037 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,039 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:51,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:51,039 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,040 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,040 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:51,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:51,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:51,042 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:51,042 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:51,042 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:51,042 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:51,042 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:51,042 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:51,042 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:51,042 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:51,043 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:51,043 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:51,043 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:51,043 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,043 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:51,043 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:51,049 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:51,049 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:51,049 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,050 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:51,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:51,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64f60b7[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:51,051 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:51,051 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,053 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,054 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:51,054 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:51,054 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:51,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:51,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:51,059 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:51,059 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,059 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:51,059 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:51,059 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:51,059 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:51,059 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:51,059 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:51,059 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:51,063 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,063 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:51,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,063 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:51,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,065 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:51,066 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,071 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:51,071 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:51,071 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,071 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,071 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:51,071 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,071 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,071 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,071 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,071 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:51,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:51,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:51,072 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:51,072 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:51,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:51,080 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/dd72b0e248204264a910e6bb5ab4db21 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/dd72b0e248204264a910e6bb5ab4db21 2023-04-18 22:55:51,087 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/C of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into dd72b0e248204264a910e6bb5ab4db21(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:51,087 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:51,087 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/C, priority=13, startTime=1681858550555; duration=0sec 2023-04-18 22:55:51,087 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:51,118 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/d1261bb3e9374f28aa0da63585b23078 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/d1261bb3e9374f28aa0da63585b23078 2023-04-18 22:55:51,124 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/B of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into d1261bb3e9374f28aa0da63585b23078(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:51,124 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:51,124 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/B, priority=13, startTime=1681858550555; duration=0sec 2023-04-18 22:55:51,125 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:51,156 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/b4ec40b2807048348f17a2be3d97a567 2023-04-18 22:55:51,172 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:51,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:51,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:51,175 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/9e329bc036b04f68b9055c32c5d49cd2 2023-04-18 22:55:51,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:51,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:51,178 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:51,178 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:51,178 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:51,178 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:51,178 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,178 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,180 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,180 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:51,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:51,180 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,180 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:51,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/3358aee613dd4bafa88201be8b0c4df1 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3358aee613dd4bafa88201be8b0c4df1 2023-04-18 22:55:51,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3358aee613dd4bafa88201be8b0c4df1, entries=200, sequenceid=476, filesize=14.3 K 2023-04-18 22:55:51,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/b4ec40b2807048348f17a2be3d97a567 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/b4ec40b2807048348f17a2be3d97a567 2023-04-18 22:55:51,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:51,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:51,187 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:51,187 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:51,187 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:51,190 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:51,190 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:51,190 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:51,190 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:51,190 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:51,190 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:51,190 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:51,193 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/b4ec40b2807048348f17a2be3d97a567, entries=150, sequenceid=476, filesize=11.9 K 2023-04-18 22:55:51,193 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/9e329bc036b04f68b9055c32c5d49cd2 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/9e329bc036b04f68b9055c32c5d49cd2 2023-04-18 22:55:51,194 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,194 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:51,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:51,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58d8d3e0[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:51,197 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:51,197 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,198 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/9e329bc036b04f68b9055c32c5d49cd2, entries=150, sequenceid=476, filesize=11.9 K 2023-04-18 22:55:51,198 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 550ms, sequenceid=476, compaction requested=true 2023-04-18 22:55:51,199 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:51,199 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:51,199 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:55:51,199 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:51,199 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-18 22:55:51,199 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:51,200 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-18 22:55:51,200 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-18 22:55:51,200 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. because compaction request was cancelled 2023-04-18 22:55:51,200 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-18 22:55:51,201 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53864 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:55:51,201 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/A is initiating minor compaction (all files) 2023-04-18 22:55:51,201 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/A in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:51,201 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/0c6ac61f183b41fbb7df8298f74cd6a9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/4188da27a08348fcaee36508088a40c5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/05b6a1c18d9a499293acee7fba3a3e1f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3358aee613dd4bafa88201be8b0c4df1] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=52.6 K 2023-04-18 22:55:51,201 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,202 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:51,202 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:51,202 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:51,202 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,202 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:51,203 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 0c6ac61f183b41fbb7df8298f74cd6a9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681858548413 2023-04-18 22:55:51,203 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,203 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:51,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,203 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-18 22:55:51,203 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-18 22:55:51,203 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. because compaction request was cancelled 2023-04-18 22:55:51,203 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:51,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:51,204 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 4188da27a08348fcaee36508088a40c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681858549683 2023-04-18 22:55:51,203 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:51,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:51,204 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 05b6a1c18d9a499293acee7fba3a3e1f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1681858550362 2023-04-18 22:55:51,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,205 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 3358aee613dd4bafa88201be8b0c4df1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1681858550502 2023-04-18 22:55:51,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:51,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:51,208 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:51,208 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:51,208 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:51,208 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:51,208 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:51,210 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,210 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:51,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,210 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:51,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:51,211 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,214 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:51,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,220 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#A#compaction#454 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:51,228 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:51,228 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:51,228 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:51,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:51,228 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:51,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:51,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:51,229 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,229 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:51,229 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,229 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:51,229 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,229 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,229 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,229 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:51,324 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:55:51,325 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:51,325 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:51,325 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:51,325 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:51,325 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:51,325 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:51,329 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:51,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:51,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:51,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:51,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:51,355 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:51,355 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:51,355 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:51,355 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:51,355 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,355 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,357 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,357 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:51,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:51,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:51,357 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,357 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:51,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:51,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:51,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:51,367 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:51,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:51,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:51,380 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:51,380 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:51,380 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:51,380 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:51,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:51,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:51,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,381 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:51,381 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:51,382 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,382 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:51,382 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,382 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:51,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:51,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32dc1c8d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:51,384 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:51,384 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,386 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,386 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,386 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:51,386 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:51,386 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:51,386 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:51,393 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:51,394 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:51,394 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:51,394 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:51,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:51,396 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:51,396 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:51,396 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:51,396 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:51,396 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:51,398 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,399 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,399 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:51,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,400 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,402 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,405 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:51,405 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:51,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:51,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:51,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:51,405 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,406 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,406 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,406 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,406 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,406 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,406 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,406 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:51,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:51,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:51,406 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:51,406 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:51,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858551368, max timestamp=1681858551414Num uniques 859; 2023-04-18 22:55:51,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858551368, max timestamp=1681858551414Num uniques 859; 2023-04-18 22:55:51,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:55:51,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858551368, max timestamp=1681858551414Num uniques 859; 2023-04-18 22:55:51,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:55:51,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:55:51,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:51,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858611467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:51,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858611473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:51,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858611474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,506 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:51,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:51,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:51,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:51,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:51,516 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:51,516 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:51,516 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:51,516 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:51,517 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,517 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,519 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:51,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:51,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,520 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,520 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:51,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:51,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:51,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:51,528 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:51,528 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:51,528 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:51,528 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:51,531 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,531 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:51,533 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,533 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:51,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:51,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5569b2f[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:51,535 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:51,535 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,537 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,538 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:51,538 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:51,538 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:51,538 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,538 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:51,539 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:51,539 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:51,539 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:51,539 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:51,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:51,542 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:51,542 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:51,542 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:51,542 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:51,542 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:51,544 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,544 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,544 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:51,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:51,546 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,549 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,554 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:51,554 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:51,554 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,554 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,554 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,554 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,554 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:51,554 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,554 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:51,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:51,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:51,555 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:51,555 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:51,554 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:51,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:51,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858611579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:51,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858611591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:51,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858611591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,643 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/77a9a75c8f8949c78f02b08e02f679fd as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/77a9a75c8f8949c78f02b08e02f679fd 2023-04-18 22:55:51,648 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/A of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into 77a9a75c8f8949c78f02b08e02f679fd(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:51,648 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:51,648 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/A, priority=12, startTime=1681858551199; duration=0sec 2023-04-18 22:55:51,648 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:51,655 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:51,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:51,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:51,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:51,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:51,660 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:51,660 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:51,660 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:51,660 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:51,660 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,661 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,662 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:51,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:51,662 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,662 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,662 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:51,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:51,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:51,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:51,663 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:51,663 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:51,663 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:51,668 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:51,668 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:51,668 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:51,668 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:51,668 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:51,668 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:51,668 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:51,668 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:51,668 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,668 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:51,668 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:51,670 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:51,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:51,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@786109ea[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:51,676 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:51,676 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,680 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:51,680 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,681 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:51,682 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:51,685 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:51,685 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:51,685 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:51,685 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:51,685 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:51,687 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:51,688 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,688 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:51,688 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,689 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,692 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:51,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:51,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:51,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:51,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:51,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:51,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:51,692 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:51,693 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:51,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:51,793 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:51,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:51,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:51,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858611792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:51,796 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/f9b75e6f76574537a3e5d87abb596f46 2023-04-18 22:55:51,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:51,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:51,796 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:51,796 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:51,796 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:51,796 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:51,797 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,797 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,800 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,800 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,800 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:51,804 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:51,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:51,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:51,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858611799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:51,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:51,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:51,806 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:51,806 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:51,806 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:51,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:51,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858611804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:51,817 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:51,819 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:51,819 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:51,819 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:51,819 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:51,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:51,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:51,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:51,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66b892d5[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:51,823 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:51,823 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,824 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,824 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:51,824 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,824 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:51,824 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:51,825 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:51,826 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:51,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:51,829 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:51,829 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:51,829 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:51,829 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:51,829 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:51,835 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,835 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:51,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,835 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:51,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:51,837 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,838 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,842 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:51,842 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,842 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,842 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,842 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,842 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:51,842 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,842 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:51,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:51,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:51,842 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:51,842 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:51,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:51,843 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:51,843 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:51,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:51,852 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/e43266fb897b48819693e29a5c2b55e7 2023-04-18 22:55:51,905 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/73239a68d87143beae56054ace9ad420 2023-04-18 22:55:51,909 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/f9b75e6f76574537a3e5d87abb596f46 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f9b75e6f76574537a3e5d87abb596f46 2023-04-18 22:55:51,913 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f9b75e6f76574537a3e5d87abb596f46, entries=200, sequenceid=488, filesize=14.3 K 2023-04-18 22:55:51,914 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/e43266fb897b48819693e29a5c2b55e7 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/e43266fb897b48819693e29a5c2b55e7 2023-04-18 22:55:51,943 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:51,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:51,945 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/e43266fb897b48819693e29a5c2b55e7, entries=150, sequenceid=488, filesize=11.9 K 2023-04-18 22:55:51,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:51,946 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/73239a68d87143beae56054ace9ad420 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/73239a68d87143beae56054ace9ad420 2023-04-18 22:55:51,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:51,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:51,956 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:51,956 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:51,956 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:51,956 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:51,964 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,964 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,965 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:51,965 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:51,966 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:51,966 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:51,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:51,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:51,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:51,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:51,976 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/73239a68d87143beae56054ace9ad420, entries=150, sequenceid=488, filesize=11.9 K 2023-04-18 22:55:51,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:51,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:51,977 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 653ms, sequenceid=488, compaction requested=true 2023-04-18 22:55:51,977 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:51,977 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:51,977 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:51,977 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-18 22:55:51,977 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:51,977 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-18 22:55:51,978 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-18 22:55:51,978 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-18 22:55:51,978 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. because compaction request was cancelled 2023-04-18 22:55:51,978 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:55:51,979 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:51,980 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/C is initiating minor compaction (all files) 2023-04-18 22:55:51,980 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/C in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:51,980 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/dd72b0e248204264a910e6bb5ab4db21, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/9e329bc036b04f68b9055c32c5d49cd2, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/73239a68d87143beae56054ace9ad420] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=36.0 K 2023-04-18 22:55:51,980 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:55:51,980 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/B is initiating minor compaction (all files) 2023-04-18 22:55:51,980 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/B in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:51,980 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/d1261bb3e9374f28aa0da63585b23078, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/b4ec40b2807048348f17a2be3d97a567, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/e43266fb897b48819693e29a5c2b55e7] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=36.0 K 2023-04-18 22:55:51,981 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting dd72b0e248204264a910e6bb5ab4db21, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1681858550366 2023-04-18 22:55:51,981 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting d1261bb3e9374f28aa0da63585b23078, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1681858550366 2023-04-18 22:55:51,981 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 9e329bc036b04f68b9055c32c5d49cd2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1681858550502 2023-04-18 22:55:51,981 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting b4ec40b2807048348f17a2be3d97a567, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1681858550502 2023-04-18 22:55:51,982 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 73239a68d87143beae56054ace9ad420, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681858550651 2023-04-18 22:55:51,982 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting e43266fb897b48819693e29a5c2b55e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681858550651 2023-04-18 22:55:51,992 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:51,992 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:51,992 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:51,992 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:51,992 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-18 22:55:51,992 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:51,992 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:51,993 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:51,993 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:51,993 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:51,993 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:52,001 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#B#compaction#458 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:52,011 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#C#compaction#459 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:52,032 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/732e960ea8754019a2b949374ec2454e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/732e960ea8754019a2b949374ec2454e 2023-04-18 22:55:52,035 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/B of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into 732e960ea8754019a2b949374ec2454e(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:52,035 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:52,035 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/B, priority=13, startTime=1681858551977; duration=0sec 2023-04-18 22:55:52,036 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:52,046 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/2af1eea84d9f46668b6f97970bfefa7f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/2af1eea84d9f46668b6f97970bfefa7f 2023-04-18 22:55:52,050 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/C of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into 2af1eea84d9f46668b6f97970bfefa7f(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:52,050 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:52,050 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/C, priority=13, startTime=1681858551977; duration=0sec 2023-04-18 22:55:52,050 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:52,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:52,101 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:52,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:52,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858612121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:52,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858612128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:52,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858612132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:52,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858612232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:52,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858612240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:52,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858612241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,438 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/3be8684f6bb946479e2eaa7f6756716b 2023-04-18 22:55:52,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:52,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858612440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,458 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/6fde858b949c439680289c6282ed425a 2023-04-18 22:55:52,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:52,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858612452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:52,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858612452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:52,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858612750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:52,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858612764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:52,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858612764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,878 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/daa72ca4122f417ca9020be108e6ee35 2023-04-18 22:55:52,882 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/3be8684f6bb946479e2eaa7f6756716b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3be8684f6bb946479e2eaa7f6756716b 2023-04-18 22:55:52,889 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3be8684f6bb946479e2eaa7f6756716b, entries=150, sequenceid=525, filesize=11.9 K 2023-04-18 22:55:52,890 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/6fde858b949c439680289c6282ed425a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/6fde858b949c439680289c6282ed425a 2023-04-18 22:55:52,894 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/6fde858b949c439680289c6282ed425a, entries=150, sequenceid=525, filesize=11.9 K 2023-04-18 22:55:52,894 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/daa72ca4122f417ca9020be108e6ee35 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/daa72ca4122f417ca9020be108e6ee35 2023-04-18 22:55:52,898 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/daa72ca4122f417ca9020be108e6ee35, entries=150, sequenceid=525, filesize=11.9 K 2023-04-18 22:55:52,899 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 907ms, sequenceid=525, compaction requested=true 2023-04-18 22:55:52,899 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:52,899 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:52,899 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:52,899 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:52,899 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,899 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:52,899 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:52,900 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,900 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:52,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:52,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:52,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:52,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34d2c2fb[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:52,905 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:52,905 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:52,907 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:52,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:52,907 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:52,907 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:52,907 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,907 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:52,907 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:52,908 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:52,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:52,909 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:52,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:52,911 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:52,911 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:52,911 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:52,911 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:52,911 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:52,913 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:52,913 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:52,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:52,913 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:52,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:52,913 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:52,914 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,916 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,916 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:52,916 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:52,916 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:52,916 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:52,916 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:52,916 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:52,916 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:52,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:52,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:52,917 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:52,917 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:52,917 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:52,917 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:52,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:52,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:52,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:53,017 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:53,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:53,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:53,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:53,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:53,022 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:53,022 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:53,022 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:53,022 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:53,023 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,023 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,027 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,027 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:53,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:53,028 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,028 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:53,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:53,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:53,029 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:53,029 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:53,029 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:53,036 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:53,036 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:53,036 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:53,036 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:53,036 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:55:53,036 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:53,037 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:53,037 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:53,037 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:53,037 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:53,037 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:53,059 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/61a0c2e0b89a44e5b3000b09b63f9003 2023-04-18 22:55:53,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:53,259 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:53,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:53,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858613312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858553303, max timestamp=1681858553317Num uniques 859; 2023-04-18 22:55:53,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:55:53,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858553303, max timestamp=1681858553316Num uniques 859; 2023-04-18 22:55:53,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:55:53,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858553303, max timestamp=1681858553317Num uniques 859; 2023-04-18 22:55:53,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:55:53,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:53,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858613399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:53,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858613402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:53,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858613421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,483 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/fb48a873b2544956b1bf630c892243e3 2023-04-18 22:55:53,494 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/c6cfee8a95e3472fb3632864aa01b629 2023-04-18 22:55:53,498 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/61a0c2e0b89a44e5b3000b09b63f9003 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/61a0c2e0b89a44e5b3000b09b63f9003 2023-04-18 22:55:53,511 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/61a0c2e0b89a44e5b3000b09b63f9003, entries=150, sequenceid=537, filesize=11.9 K 2023-04-18 22:55:53,513 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/fb48a873b2544956b1bf630c892243e3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/fb48a873b2544956b1bf630c892243e3 2023-04-18 22:55:53,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:53,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858613512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,524 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/fb48a873b2544956b1bf630c892243e3, entries=150, sequenceid=537, filesize=11.9 K 2023-04-18 22:55:53,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:53,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858613515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,525 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/c6cfee8a95e3472fb3632864aa01b629 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/c6cfee8a95e3472fb3632864aa01b629 2023-04-18 22:55:53,529 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/c6cfee8a95e3472fb3632864aa01b629, entries=150, sequenceid=537, filesize=11.9 K 2023-04-18 22:55:53,530 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 494ms, sequenceid=537, compaction requested=true 2023-04-18 22:55:53,530 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:53,530 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:53,530 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:53,530 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:53,530 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,530 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:53,530 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:53,533 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:53,533 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:53,533 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,533 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:53,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:53,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ecb8ad[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:53,535 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:53,535 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,536 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,536 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:53,536 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:53,536 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:53,536 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,537 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:53,538 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:53,538 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:53,538 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:53,539 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:53,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:53,543 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:53,543 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:53,543 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:53,543 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:53,543 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:53,548 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,548 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:53,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,548 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:53,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:53,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,549 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:53,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,552 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,554 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:53,554 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:53,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:53,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:53,555 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,555 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:53,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:53,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:53,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:53,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:53,555 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,555 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,555 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,555 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:53,555 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:53,555 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,556 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,556 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858553386, max timestamp=1681858553637Num uniques 1018; 2023-04-18 22:55:53,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858553386, max timestamp=1681858553637Num uniques 1018; 2023-04-18 22:55:53,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-18 22:55:53,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-18 22:55:53,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858553386, max timestamp=1681858553637Num uniques 1018; 2023-04-18 22:55:53,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-18 22:55:53,656 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:53,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:53,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:53,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:53,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:53,660 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:53,660 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:53,660 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:53,660 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:53,662 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,662 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:53,663 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:53,664 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,664 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:53,666 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=53.67 KB heapSize=145.08 KB 2023-04-18 22:55:53,666 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:53,666 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:53,666 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:53,666 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:53,666 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:53,666 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:53,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:53,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:53,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:53,676 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:53,685 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:53,685 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:53,685 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:53,687 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:53,687 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:53,687 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:53,687 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:53,688 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:53,688 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,688 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:53,688 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:53,689 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:53,690 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,691 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,691 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:53,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:53,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75b2cfa3[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:53,691 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:53,692 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,695 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,695 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:53,695 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:53,695 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:53,696 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,696 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:53,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:53,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:53,701 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:53,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:53,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:53,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:53,701 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:53,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:53,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:53,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:53,704 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:53,705 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,710 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:53,710 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,710 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:53,710 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:53,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:53,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:53,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:53,710 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:53,710 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,710 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,710 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,710 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,710 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:53,711 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:53,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:53,711 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:53,712 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/f348df132da0402b92dec3408781c94e 2023-04-18 22:55:53,735 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/e1cc47c7e2784ad7a0e14f7e4612226e 2023-04-18 22:55:53,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858553743, max timestamp=1681858553771Num uniques 859; 2023-04-18 22:55:53,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858553743, max timestamp=1681858553771Num uniques 859; 2023-04-18 22:55:53,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858553743, max timestamp=1681858553771Num uniques 859; 2023-04-18 22:55:53,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:55:53,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:55:53,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:55:53,811 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:53,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:53,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:53,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:53,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:53,814 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:53,814 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:53,814 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:53,814 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:53,815 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,815 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,817 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,817 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:53,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:53,817 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,817 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:53,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:53,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:53,819 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:53,819 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:53,819 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:53,819 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:53,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,821 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:53,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,821 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:53,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57bed5fa[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:53,823 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:53,823 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,824 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:53,824 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,824 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:53,825 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:53,825 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:53,825 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,825 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:53,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:53,828 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:53,829 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:53,829 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:53,829 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:53,829 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:53,831 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,831 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:53,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,831 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:53,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:53,831 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:53,833 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,835 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,835 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,835 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,835 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,835 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,836 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:53,836 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:53,836 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,836 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,836 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:53,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:53,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:53,836 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:53,836 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:53,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:53,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:53,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:53,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:53,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858613820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:53,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858613837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:53,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858613838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,937 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:53,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:53,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:53,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:53,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:53,942 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:53,942 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:53,942 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:53,942 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:53,942 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,943 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,944 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:53,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:53,944 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:53,944 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,944 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:53,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:53,946 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:53,946 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:53,946 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:53,946 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:53,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:53,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858613939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,948 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:53,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858613947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:53,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858613948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:53,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755900ec[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:53,950 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:53,951 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,953 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,954 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:53,954 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:53,954 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:53,954 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,954 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:53,955 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,955 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:53,955 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:53,955 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:53,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:53,959 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:53,959 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:53,959 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:53,959 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:53,959 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:53,961 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,961 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,961 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:53,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:53,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:53,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,963 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:53,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,965 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:53,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:53,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,971 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:53,971 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,971 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:53,971 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:53,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:53,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:53,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:53,971 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:53,971 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:53,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:53,972 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:53,972 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:53,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:53,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:53,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:53,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,072 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:54,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:54,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:54,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:54,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:54,077 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:54,077 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:54,077 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:54,077 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:54,077 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,077 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,078 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:54,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:54,079 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,079 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,079 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:54,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:54,081 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:54,081 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:54,081 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:54,081 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:54,081 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:54,081 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:54,082 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:54,082 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:54,082 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:54,082 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:54,082 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:54,082 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,082 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:54,082 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:54,083 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,083 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,084 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,084 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1510f73a[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:54,085 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:54,085 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,086 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,086 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:54,086 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:54,086 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:54,087 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,087 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:54,089 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,090 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:54,090 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,090 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:54,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:54,093 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:54,093 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:54,093 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:54,093 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:54,093 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:54,096 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,096 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,096 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:54,097 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,098 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,101 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:54,101 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:54,101 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,101 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:54,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:54,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:54,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:54,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:54,101 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,101 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,101 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,101 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,101 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,101 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,101 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:54,101 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:54,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:54,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:54,146 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/288cff72243a468bba66334ae25a9d5d 2023-04-18 22:55:54,150 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/f348df132da0402b92dec3408781c94e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f348df132da0402b92dec3408781c94e 2023-04-18 22:55:54,154 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f348df132da0402b92dec3408781c94e, entries=200, sequenceid=579, filesize=14.3 K 2023-04-18 22:55:54,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/e1cc47c7e2784ad7a0e14f7e4612226e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/e1cc47c7e2784ad7a0e14f7e4612226e 2023-04-18 22:55:54,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:54,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858614148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,159 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/e1cc47c7e2784ad7a0e14f7e4612226e, entries=150, sequenceid=579, filesize=11.9 K 2023-04-18 22:55:54,159 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/288cff72243a468bba66334ae25a9d5d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/288cff72243a468bba66334ae25a9d5d 2023-04-18 22:55:54,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:54,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858614151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:54,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858614151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,163 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/288cff72243a468bba66334ae25a9d5d, entries=150, sequenceid=579, filesize=11.9 K 2023-04-18 22:55:54,164 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.03 KB/148512, currentSize=134.18 KB/137400 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 498ms, sequenceid=579, compaction requested=true 2023-04-18 22:55:54,164 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:54,164 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:54,164 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 22:55:54,164 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:55:54,164 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:54,164 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:54,165 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49188 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:55:54,166 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/B is initiating minor compaction (all files) 2023-04-18 22:55:54,166 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/B in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:54,166 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/732e960ea8754019a2b949374ec2454e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/6fde858b949c439680289c6282ed425a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/fb48a873b2544956b1bf630c892243e3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/e1cc47c7e2784ad7a0e14f7e4612226e] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=48.0 K 2023-04-18 22:55:54,166 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66186 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 22:55:54,166 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/A is initiating minor compaction (all files) 2023-04-18 22:55:54,166 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/A in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:54,166 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/77a9a75c8f8949c78f02b08e02f679fd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f9b75e6f76574537a3e5d87abb596f46, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3be8684f6bb946479e2eaa7f6756716b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/61a0c2e0b89a44e5b3000b09b63f9003, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f348df132da0402b92dec3408781c94e] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=64.6 K 2023-04-18 22:55:54,167 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 732e960ea8754019a2b949374ec2454e, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681858550651 2023-04-18 22:55:54,167 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 77a9a75c8f8949c78f02b08e02f679fd, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1681858550502 2023-04-18 22:55:54,167 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 6fde858b949c439680289c6282ed425a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1681858551456 2023-04-18 22:55:54,168 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting fb48a873b2544956b1bf630c892243e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1681858552113 2023-04-18 22:55:54,168 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting f9b75e6f76574537a3e5d87abb596f46, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681858550651 2023-04-18 22:55:54,170 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting e1cc47c7e2784ad7a0e14f7e4612226e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1681858553397 2023-04-18 22:55:54,170 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 3be8684f6bb946479e2eaa7f6756716b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1681858551456 2023-04-18 22:55:54,171 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 61a0c2e0b89a44e5b3000b09b63f9003, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1681858552113 2023-04-18 22:55:54,171 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting f348df132da0402b92dec3408781c94e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1681858553397 2023-04-18 22:55:54,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:54,181 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-18 22:55:54,181 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:54,181 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:54,182 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:54,182 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:54,182 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:54,182 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:55:54,183 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#B#compaction#469 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:54,186 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#A#compaction#470 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:54,201 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:54,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:54,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:54,205 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/200d3586dd91409eb02ad18b1d79371b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/200d3586dd91409eb02ad18b1d79371b 2023-04-18 22:55:54,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:54,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:54,212 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:54,212 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:54,212 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:54,212 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:54,213 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,213 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,213 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/1bdbad68c99d4acab32bd99d01cd9af0 2023-04-18 22:55:54,214 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,214 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:54,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:54,214 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/A of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into 200d3586dd91409eb02ad18b1d79371b(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:54,214 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:54,215 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/A, priority=11, startTime=1681858554164; duration=0sec 2023-04-18 22:55:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,215 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:54,215 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:55:54,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,215 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,216 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:54,218 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49188 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:55:54,218 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/C is initiating minor compaction (all files) 2023-04-18 22:55:54,218 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/C in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:54,218 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/2af1eea84d9f46668b6f97970bfefa7f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/daa72ca4122f417ca9020be108e6ee35, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/c6cfee8a95e3472fb3632864aa01b629, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/288cff72243a468bba66334ae25a9d5d] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=48.0 K 2023-04-18 22:55:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:54,219 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 2af1eea84d9f46668b6f97970bfefa7f, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681858550651 2023-04-18 22:55:54,219 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting daa72ca4122f417ca9020be108e6ee35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1681858551456 2023-04-18 22:55:54,219 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting c6cfee8a95e3472fb3632864aa01b629, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1681858552113 2023-04-18 22:55:54,220 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 288cff72243a468bba66334ae25a9d5d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1681858553397 2023-04-18 22:55:54,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:54,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:54,220 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:54,220 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:54,220 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:54,228 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:54,228 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:54,228 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:54,228 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:54,228 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:54,228 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:54,228 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:54,228 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:54,228 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:54,231 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:54,232 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#C#compaction#472 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:54,233 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,234 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,234 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:54,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:54,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@763cedec[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:54,241 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:54,241 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,242 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,242 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:54,242 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:54,242 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:54,242 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:54,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48912 deadline: 1681858614235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,242 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:54,244 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:54,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:54,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48868 deadline: 1681858614236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:54,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:54,248 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:54,248 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:54,248 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:54,248 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:54,248 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:54,249 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,249 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,250 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:54,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:54,250 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,252 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,255 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/2438fef9ecae42479014585888985e45 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/2438fef9ecae42479014585888985e45 2023-04-18 22:55:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,257 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,257 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,257 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,257 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:54,257 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:54,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:54,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:54,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:54,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:54,258 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:54,258 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:54,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:54,257 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,258 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,258 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,258 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:54,258 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:54,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:54,261 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/C of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into 2438fef9ecae42479014585888985e45(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:54,261 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:54,261 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/C, priority=12, startTime=1681858554164; duration=0sec 2023-04-18 22:55:54,261 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:54,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:54,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48912 deadline: 1681858614344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:54,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48868 deadline: 1681858614346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,358 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:54,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:54,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:54,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:54,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:54,362 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:54,362 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:54,363 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:54,363 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:54,363 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,363 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,365 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:54,365 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:54,365 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,365 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:54,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:54,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:54,366 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:54,366 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:54,366 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:54,367 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:54,367 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:54,367 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:54,367 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:54,367 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:54,368 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,368 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,369 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,369 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:54,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@578ce7d4[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:54,374 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:54,374 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,375 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,376 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:54,376 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:54,376 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:54,376 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,376 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:54,378 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:54,378 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:54,378 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:54,379 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:54,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:54,383 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:54,383 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:54,383 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:54,383 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:54,383 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:54,385 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,385 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,385 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:54,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:54,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:54,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,387 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,388 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,398 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,398 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,398 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:54,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:54,398 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:54,398 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:54,399 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:54,398 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,399 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:54,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:54,399 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,399 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,399 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:54,399 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:54,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:54,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858614456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:54,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:54,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858614462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858614463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,499 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:54,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:54,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:54,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:54,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:54,503 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:54,503 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:54,503 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:54,503 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:54,503 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,503 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,506 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:54,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:54,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:54,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:54,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:54,514 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:54,514 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:54,514 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:54,514 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:54,514 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:54,516 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,516 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,516 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,516 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:54,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6fd3a9[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:54,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:54,520 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,521 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:54,521 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:54,522 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,522 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:54,524 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:54,524 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:54,524 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:54,524 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:54,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:54,529 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:54,529 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:54,529 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:54,529 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:54,529 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:54,534 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,534 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,534 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:54,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:54,535 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,537 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,539 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:54,539 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:54,539 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:54,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:54,539 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:54,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:54,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:54,539 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,539 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:54,539 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:54,539 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,540 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,540 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,540 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,540 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:54,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:54,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48868 deadline: 1681858614549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:54,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48912 deadline: 1681858614550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,604 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/85e3b6e0f2ef449bb60b1d59692bc17d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/85e3b6e0f2ef449bb60b1d59692bc17d 2023-04-18 22:55:54,618 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/B of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into 85e3b6e0f2ef449bb60b1d59692bc17d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:54,618 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:54,618 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/B, priority=12, startTime=1681858554164; duration=0sec 2023-04-18 22:55:54,618 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:54,638 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/784513793b9b42888729972dbf484a12 2023-04-18 22:55:54,640 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:54,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:54,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:54,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:54,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:54,650 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:54,650 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:54,650 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:54,650 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:54,650 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,650 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,652 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,652 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:54,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:54,652 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,652 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:54,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:54,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:54,659 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:54,659 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:54,659 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:54,660 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:54,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:54,661 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:54,661 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:54,661 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:54,661 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:54,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:54,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:54,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:54,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:54,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,663 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:54,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@785440f9[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:54,665 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:54,665 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,666 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,666 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:54,666 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:54,666 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:54,667 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,667 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:54,668 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,669 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:54,669 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:54,669 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:54,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:54,672 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:54,672 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:54,672 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:54,672 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:54,672 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:54,674 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,674 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,674 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,674 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,676 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,678 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,678 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,678 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,678 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:54,678 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:54,678 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,678 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,678 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:54,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:54,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:54,679 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:54,679 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:54,678 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:54,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:54,678 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:54,779 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:54,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:54,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:54,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:54,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:54,782 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:54,782 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:54,783 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:54,783 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:54,783 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,783 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,784 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:54,784 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:54,785 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,785 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:54,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:54,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:54,786 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:54,786 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:54,786 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:54,787 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:54,787 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:54,787 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:54,787 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:54,787 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:54,787 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:54,787 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:54,787 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:54,787 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,787 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:54,787 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:54,790 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,790 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,790 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,790 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:54,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69169192[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:54,794 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:54,794 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,795 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,796 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,796 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:54,796 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:54,796 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:54,796 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:54,797 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:54,797 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:54,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:54,803 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:54,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:54,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:54,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:54,803 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:54,804 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,804 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,804 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:54,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,805 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,806 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,812 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,812 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,812 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,812 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,812 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,812 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,812 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,812 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:54,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:54,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:54,812 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:54,812 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:54,813 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:54,813 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:54,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:54,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:54,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:54,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:54,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48868 deadline: 1681858614855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:54,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48912 deadline: 1681858614856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,913 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:54,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:54,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:54,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:54,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:54,916 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:54,916 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:54,916 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:54,916 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:54,917 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,917 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,918 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:54,918 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:54,919 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,919 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:54,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:54,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:54,920 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:54,920 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:54,920 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:54,921 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:54,921 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:54,921 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:54,921 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:54,921 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:54,921 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:54,921 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:54,921 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:54,921 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,921 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:54,921 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:54,925 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,926 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,926 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,926 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:54,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59946923[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:54,927 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:54,927 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,933 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,933 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:54,933 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:54,933 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:54,933 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,933 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:54,935 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,935 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,935 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:54,935 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:54,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:54,937 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:54,937 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:54,937 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:54,937 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:54,937 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:54,939 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,939 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:54,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,939 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:54,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:54,940 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,941 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:54,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,943 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,943 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,943 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:54,943 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,944 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,944 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:54,944 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:54,944 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:54,944 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:54,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:54,944 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:54,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:54,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:54,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:54,944 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:54,944 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:54,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:54,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858614966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:54,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858614972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:54,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:54,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858614972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,044 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:55,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:55,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:55,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:55,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:55,049 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:55,049 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:55,049 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:55,049 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:55,049 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:55,049 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:55,056 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:55,056 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:55,056 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,057 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:55,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:55,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:55,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:55,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:55,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:55,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:55,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:55,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:55,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:55,058 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:55,058 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:55,058 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:55,058 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:55,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:55,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:55,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:55,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:55,060 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,060 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:55,060 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:55,060 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,062 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/9709edbacedf4a2099565e2acdec7eb3 2023-04-18 22:55:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:55,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:55,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4df20b23[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:55,062 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:55,062 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:55,064 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:55,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:55,064 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:55,064 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:55,064 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:55,066 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,066 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:55,067 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/1bdbad68c99d4acab32bd99d01cd9af0 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/1bdbad68c99d4acab32bd99d01cd9af0 2023-04-18 22:55:55,068 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,068 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:55,068 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:55,068 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:55,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:55,071 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:55,071 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:55,071 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:55,071 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:55,071 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:55,071 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/1bdbad68c99d4acab32bd99d01cd9af0, entries=200, sequenceid=615, filesize=14.3 K 2023-04-18 22:55:55,072 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/784513793b9b42888729972dbf484a12 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/784513793b9b42888729972dbf484a12 2023-04-18 22:55:55,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:55,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:55,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:55,075 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:55,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:55,076 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:55,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:55,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,077 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,079 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/784513793b9b42888729972dbf484a12, entries=150, sequenceid=615, filesize=11.9 K 2023-04-18 22:55:55,080 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/9709edbacedf4a2099565e2acdec7eb3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/9709edbacedf4a2099565e2acdec7eb3 2023-04-18 22:55:55,080 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,080 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:55,080 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:55,080 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,080 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:55,080 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:55,080 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:55,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:55,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:55,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:55,081 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:55,080 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:55,081 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:55,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:55,082 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:55,082 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:55,085 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/9709edbacedf4a2099565e2acdec7eb3, entries=150, sequenceid=615, filesize=11.9 K 2023-04-18 22:55:55,085 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 904ms, sequenceid=615, compaction requested=false 2023-04-18 22:55:55,085 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:55,182 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:55,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:55,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:55:55,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:55:55,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:55:55,185 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:55:55,185 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:55,185 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:55:55,185 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:55:55,186 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:55,186 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:55,187 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,187 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:55,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:55,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:55,188 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,188 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:55,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:55,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:55:55,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:55,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:55:55,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:55:55,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:55:55,189 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:55:55,189 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:55:55,189 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:55:55,189 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:55:55,189 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:55:55,191 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:55,191 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. started... 2023-04-18 22:55:55,191 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-18 22:55:55,191 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:55,191 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:55,192 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:55,192 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:55,192 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:55,192 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:55,211 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/c848574a42494feb9720bf12fd396f55 2023-04-18 22:55:55,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:55,366 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. as already flushing 2023-04-18 22:55:55,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858555394, max timestamp=1681858555425Num uniques 859; 2023-04-18 22:55:55,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:55:55,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858555394, max timestamp=1681858555425Num uniques 859; 2023-04-18 22:55:55,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:55:55,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858555394, max timestamp=1681858555425Num uniques 859; 2023-04-18 22:55:55,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:55:55,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:55,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48868 deadline: 1681858615495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:55,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48912 deadline: 1681858615502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:55,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48868 deadline: 1681858615603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:55,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48912 deadline: 1681858615610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,628 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/d4e09678f0f049f685c353caba87d60f 2023-04-18 22:55:55,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:55,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48868 deadline: 1681858615813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:55,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48912 deadline: 1681858615814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,862 DEBUG [Thread-2924] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x680e773e to 127.0.0.1:50825 2023-04-18 22:55:55,862 DEBUG [Thread-2924] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:55,867 DEBUG [Thread-2922] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x659c68ab to 127.0.0.1:50825 2023-04-18 22:55:55,868 DEBUG [Thread-2922] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:55,872 DEBUG [Thread-2920] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78592df0 to 127.0.0.1:50825 2023-04-18 22:55:55,872 DEBUG [Thread-2920] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:55,873 DEBUG [Thread-2918] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x067ea274 to 127.0.0.1:50825 2023-04-18 22:55:55,873 DEBUG [Thread-2918] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:55,874 DEBUG [Thread-2926] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b8f652e to 127.0.0.1:50825 2023-04-18 22:55:55,874 DEBUG [Thread-2926] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:55,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:55,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48956 deadline: 1681858615972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:55,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48880 deadline: 1681858615983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:55,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:55:55,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48892 deadline: 1681858615990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:56,038 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/18c0dc233c094c43b7cf8d4b712cee0d 2023-04-18 22:55:56,041 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/c848574a42494feb9720bf12fd396f55 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/c848574a42494feb9720bf12fd396f55 2023-04-18 22:55:56,044 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/c848574a42494feb9720bf12fd396f55, entries=100, sequenceid=627, filesize=9.5 K 2023-04-18 22:55:56,045 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/d4e09678f0f049f685c353caba87d60f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/d4e09678f0f049f685c353caba87d60f 2023-04-18 22:55:56,048 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/d4e09678f0f049f685c353caba87d60f, entries=100, sequenceid=627, filesize=9.5 K 2023-04-18 22:55:56,049 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/18c0dc233c094c43b7cf8d4b712cee0d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/18c0dc233c094c43b7cf8d4b712cee0d 2023-04-18 22:55:56,052 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/18c0dc233c094c43b7cf8d4b712cee0d, entries=100, sequenceid=627, filesize=9.5 K 2023-04-18 22:55:56,052 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 861ms, sequenceid=627, compaction requested=true 2023-04-18 22:55:56,052 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:56,052 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:55:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:55:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:55:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:55:56,054 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:55:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:56,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:55:56,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@177bc9ec[Count = 0] remaining members to acquire global barrier 2023-04-18 22:55:56,056 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:55:56,056 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:56,058 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:56,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:55:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:55:56,058 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:56,058 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:55:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:55:56,059 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:56,059 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:55:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:56,059 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:56,059 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:55:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:55:56,061 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:55:56,061 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:55:56,061 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:55:56,061 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:55:56,061 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:55:56,063 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:56,063 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:55:56,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:56,063 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:56,063 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:55:56,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:56,065 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:55:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:56,069 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:56,069 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:55:56,069 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:56,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:55:56,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:55:56,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:55:56,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:55:56,069 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:55:56,069 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:55:56,069 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:55:56,070 DEBUG [Thread-2917] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:55:56,070 DEBUG [Thread-2917] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:55:56,070 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:56,070 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:56,070 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:55:56,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:55:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:55:56,070 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:55:56,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:56,122 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=154.31 KB heapSize=440.77 KB 2023-04-18 22:55:56,122 DEBUG [Thread-2913] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51b8df2a to 127.0.0.1:50825 2023-04-18 22:55:56,122 DEBUG [Thread-2913] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:56,122 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:56,122 DEBUG [Thread-2909] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x052203cb to 127.0.0.1:50825 2023-04-18 22:55:56,122 DEBUG [Thread-2909] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:56,122 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:55:56,122 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:56,122 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:55:56,122 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:56,122 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:55:56,170 DEBUG [Thread-2917] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:55:56,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:55:56,529 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/8bef2559405e4a8eb1d8969f4d8bb4e1 2023-04-18 22:55:56,939 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/2681e280eea14814bbe2a7b4a6665e57 2023-04-18 22:55:57,348 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/551c2aecbd824e199add12c6a0b64b4d 2023-04-18 22:55:57,351 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/8bef2559405e4a8eb1d8969f4d8bb4e1 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/8bef2559405e4a8eb1d8969f4d8bb4e1 2023-04-18 22:55:57,355 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/8bef2559405e4a8eb1d8969f4d8bb4e1, entries=150, sequenceid=666, filesize=11.9 K 2023-04-18 22:55:57,355 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/2681e280eea14814bbe2a7b4a6665e57 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/2681e280eea14814bbe2a7b4a6665e57 2023-04-18 22:55:57,359 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/2681e280eea14814bbe2a7b4a6665e57, entries=150, sequenceid=666, filesize=11.9 K 2023-04-18 22:55:57,359 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/551c2aecbd824e199add12c6a0b64b4d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/551c2aecbd824e199add12c6a0b64b4d 2023-04-18 22:55:57,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/551c2aecbd824e199add12c6a0b64b4d, entries=150, sequenceid=666, filesize=11.9 K 2023-04-18 22:55:57,363 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~440.72 KB/451296, currentSize=0 B/0 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 1242ms, sequenceid=666, compaction requested=true 2023-04-18 22:55:57,363 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:57,363 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:57,363 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:55:57,363 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:57,363 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:57,363 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:55:57,364 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49286 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:55:57,365 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46880 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:55:57,365 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/A is initiating minor compaction (all files) 2023-04-18 22:55:57,365 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/B is initiating minor compaction (all files) 2023-04-18 22:55:57,365 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/A in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:57,365 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/B in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:57,365 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/200d3586dd91409eb02ad18b1d79371b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/1bdbad68c99d4acab32bd99d01cd9af0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/c848574a42494feb9720bf12fd396f55, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/8bef2559405e4a8eb1d8969f4d8bb4e1] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=48.1 K 2023-04-18 22:55:57,365 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/85e3b6e0f2ef449bb60b1d59692bc17d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/784513793b9b42888729972dbf484a12, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/d4e09678f0f049f685c353caba87d60f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/2681e280eea14814bbe2a7b4a6665e57] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=45.8 K 2023-04-18 22:55:57,365 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 200d3586dd91409eb02ad18b1d79371b, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1681858553397 2023-04-18 22:55:57,365 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 85e3b6e0f2ef449bb60b1d59692bc17d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1681858553397 2023-04-18 22:55:57,366 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 1bdbad68c99d4acab32bd99d01cd9af0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1681858553818 2023-04-18 22:55:57,366 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 784513793b9b42888729972dbf484a12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1681858553818 2023-04-18 22:55:57,366 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting c848574a42494feb9720bf12fd396f55, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1681858554232 2023-04-18 22:55:57,366 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting d4e09678f0f049f685c353caba87d60f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1681858554232 2023-04-18 22:55:57,366 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 8bef2559405e4a8eb1d8969f4d8bb4e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1681858555487 2023-04-18 22:55:57,366 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 2681e280eea14814bbe2a7b4a6665e57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1681858555487 2023-04-18 22:55:57,374 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#B#compaction#481 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:57,375 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#A#compaction#482 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:57,782 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/771431d713d1439490987ba29f8c1121 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/771431d713d1439490987ba29f8c1121 2023-04-18 22:55:57,785 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/a146a11cb883432e8393b062119d9518 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/a146a11cb883432e8393b062119d9518 2023-04-18 22:55:57,786 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/B of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into 771431d713d1439490987ba29f8c1121(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:57,786 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:57,786 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/B, priority=12, startTime=1681858557363; duration=0sec 2023-04-18 22:55:57,786 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:55:57,787 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:55:57,788 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46880 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:55:57,788 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 71e1d4dea1ebcdcdd7da8f1e7108edb0/C is initiating minor compaction (all files) 2023-04-18 22:55:57,788 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 71e1d4dea1ebcdcdd7da8f1e7108edb0/C in TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:57,788 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/2438fef9ecae42479014585888985e45, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/9709edbacedf4a2099565e2acdec7eb3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/18c0dc233c094c43b7cf8d4b712cee0d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/551c2aecbd824e199add12c6a0b64b4d] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp, totalSize=45.8 K 2023-04-18 22:55:57,788 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 2438fef9ecae42479014585888985e45, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1681858553397 2023-04-18 22:55:57,789 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 9709edbacedf4a2099565e2acdec7eb3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1681858553818 2023-04-18 22:55:57,789 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 18c0dc233c094c43b7cf8d4b712cee0d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1681858554232 2023-04-18 22:55:57,789 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/A of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into a146a11cb883432e8393b062119d9518(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:57,789 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:57,789 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 551c2aecbd824e199add12c6a0b64b4d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1681858555487 2023-04-18 22:55:57,789 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/A, priority=12, startTime=1681858557363; duration=0sec 2023-04-18 22:55:57,790 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:57,796 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 71e1d4dea1ebcdcdd7da8f1e7108edb0#C#compaction#483 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:55:57,991 DEBUG [Thread-2915] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67346c47 to 127.0.0.1:50825 2023-04-18 22:55:57,991 DEBUG [Thread-2915] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:57,993 DEBUG [Thread-2907] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15880ffb to 127.0.0.1:50825 2023-04-18 22:55:57,993 DEBUG [Thread-2907] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:58,005 DEBUG [Thread-2911] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x420f945a to 127.0.0.1:50825 2023-04-18 22:55:58,005 DEBUG [Thread-2911] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:58,005 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-18 22:55:58,005 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 163 2023-04-18 22:55:58,005 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 26 2023-04-18 22:55:58,005 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 161 2023-04-18 22:55:58,005 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 37 2023-04-18 22:55:58,005 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 173 2023-04-18 22:55:58,005 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-18 22:55:58,006 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-18 22:55:58,006 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(402): scanned 1537 2023-04-18 22:55:58,006 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(403): verified 4611 rows 2023-04-18 22:55:58,006 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(402): scanned 1540 2023-04-18 22:55:58,006 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(403): verified 4620 rows 2023-04-18 22:55:58,006 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(402): scanned 1553 2023-04-18 22:55:58,006 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(403): verified 4659 rows 2023-04-18 22:55:58,006 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(402): scanned 1558 2023-04-18 22:55:58,006 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(403): verified 4674 rows 2023-04-18 22:55:58,006 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(402): scanned 1550 2023-04-18 22:55:58,006 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(403): verified 4650 rows 2023-04-18 22:55:58,006 INFO [Listener at localhost/36651] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-18 22:55:58,006 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37705401 to 127.0.0.1:50825 2023-04-18 22:55:58,006 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:55:58,008 INFO [Listener at localhost/36651] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-18 22:55:58,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-18 22:55:58,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-18 22:55:58,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-18 22:55:58,014 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858558014"}]},"ts":"1681858558014"} 2023-04-18 22:55:58,015 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-18 22:55:58,016 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-18 22:55:58,017 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=71e1d4dea1ebcdcdd7da8f1e7108edb0, UNASSIGN}] 2023-04-18 22:55:58,019 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=71e1d4dea1ebcdcdd7da8f1e7108edb0, UNASSIGN 2023-04-18 22:55:58,019 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=71e1d4dea1ebcdcdd7da8f1e7108edb0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:55:58,019 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858558019"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858558019"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858558019"}]},"ts":"1681858558019"} 2023-04-18 22:55:58,020 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:55:58,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-18 22:55:58,172 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:58,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 71e1d4dea1ebcdcdd7da8f1e7108edb0, disabling compactions & flushes 2023-04-18 22:55:58,172 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 1 compactions to complete for region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:58,204 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/5d91cb8fe77b43549dda2cbbd0b1d5a4 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/5d91cb8fe77b43549dda2cbbd0b1d5a4 2023-04-18 22:55:58,207 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 71e1d4dea1ebcdcdd7da8f1e7108edb0/C of 71e1d4dea1ebcdcdd7da8f1e7108edb0 into 5d91cb8fe77b43549dda2cbbd0b1d5a4(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:55:58,207 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:58,207 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:58,207 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0., storeName=71e1d4dea1ebcdcdd7da8f1e7108edb0/C, priority=12, startTime=1681858557363; duration=0sec 2023-04-18 22:55:58,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:58,207 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:55:58,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. after waiting 0 ms 2023-04-18 22:55:58,208 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:58,208 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 71e1d4dea1ebcdcdd7da8f1e7108edb0 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-18 22:55:58,208 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=A 2023-04-18 22:55:58,208 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:58,208 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=B 2023-04-18 22:55:58,208 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:58,208 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 71e1d4dea1ebcdcdd7da8f1e7108edb0, store=C 2023-04-18 22:55:58,208 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:55:58,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-18 22:55:58,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/2994dd6cfbb24304a2418cb21f734340 2023-04-18 22:55:58,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-18 22:55:59,023 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/c7b122cec2554fb3a6ff01ac7aa8a200 2023-04-18 22:55:59,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-18 22:55:59,432 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/b311e211aeda468eb0e42e0fea23aecd 2023-04-18 22:55:59,436 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/A/2994dd6cfbb24304a2418cb21f734340 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/2994dd6cfbb24304a2418cb21f734340 2023-04-18 22:55:59,439 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/2994dd6cfbb24304a2418cb21f734340, entries=150, sequenceid=675, filesize=11.9 K 2023-04-18 22:55:59,439 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/B/c7b122cec2554fb3a6ff01ac7aa8a200 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/c7b122cec2554fb3a6ff01ac7aa8a200 2023-04-18 22:55:59,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/c7b122cec2554fb3a6ff01ac7aa8a200, entries=150, sequenceid=675, filesize=11.9 K 2023-04-18 22:55:59,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/.tmp/C/b311e211aeda468eb0e42e0fea23aecd as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/b311e211aeda468eb0e42e0fea23aecd 2023-04-18 22:55:59,446 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/b311e211aeda468eb0e42e0fea23aecd, entries=150, sequenceid=675, filesize=11.9 K 2023-04-18 22:55:59,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 71e1d4dea1ebcdcdd7da8f1e7108edb0 in 1239ms, sequenceid=675, compaction requested=false 2023-04-18 22:55:59,447 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/8ce97dfa6ce54bd4818b5532c1822030, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/93a24a61b66541b49e232cb01fc144bb, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/701ea3b553ed4fe1ae461b1eb1fe81ef, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/aa2b74abfc5a4d919282ab1c8cba45e0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/93a64148ceb74c11ba907545f3832b3c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/88bea1d0960d4298874becad6efacbba, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/0c6ac61f183b41fbb7df8298f74cd6a9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/0f64503b00a4427297f6690ea9436673, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/4188da27a08348fcaee36508088a40c5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/05b6a1c18d9a499293acee7fba3a3e1f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3358aee613dd4bafa88201be8b0c4df1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/77a9a75c8f8949c78f02b08e02f679fd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f9b75e6f76574537a3e5d87abb596f46, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3be8684f6bb946479e2eaa7f6756716b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/61a0c2e0b89a44e5b3000b09b63f9003, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f348df132da0402b92dec3408781c94e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/200d3586dd91409eb02ad18b1d79371b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/1bdbad68c99d4acab32bd99d01cd9af0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/c848574a42494feb9720bf12fd396f55, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/8bef2559405e4a8eb1d8969f4d8bb4e1] to archive 2023-04-18 22:55:59,448 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:55:59,449 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/8ce97dfa6ce54bd4818b5532c1822030 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/8ce97dfa6ce54bd4818b5532c1822030 2023-04-18 22:55:59,450 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/93a24a61b66541b49e232cb01fc144bb to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/93a24a61b66541b49e232cb01fc144bb 2023-04-18 22:55:59,451 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/701ea3b553ed4fe1ae461b1eb1fe81ef to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/701ea3b553ed4fe1ae461b1eb1fe81ef 2023-04-18 22:55:59,451 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/aa2b74abfc5a4d919282ab1c8cba45e0 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/aa2b74abfc5a4d919282ab1c8cba45e0 2023-04-18 22:55:59,452 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/93a64148ceb74c11ba907545f3832b3c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/93a64148ceb74c11ba907545f3832b3c 2023-04-18 22:55:59,453 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/88bea1d0960d4298874becad6efacbba to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/88bea1d0960d4298874becad6efacbba 2023-04-18 22:55:59,454 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/0c6ac61f183b41fbb7df8298f74cd6a9 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/0c6ac61f183b41fbb7df8298f74cd6a9 2023-04-18 22:55:59,454 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/0f64503b00a4427297f6690ea9436673 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/0f64503b00a4427297f6690ea9436673 2023-04-18 22:55:59,455 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/4188da27a08348fcaee36508088a40c5 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/4188da27a08348fcaee36508088a40c5 2023-04-18 22:55:59,456 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/05b6a1c18d9a499293acee7fba3a3e1f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/05b6a1c18d9a499293acee7fba3a3e1f 2023-04-18 22:55:59,457 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3358aee613dd4bafa88201be8b0c4df1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3358aee613dd4bafa88201be8b0c4df1 2023-04-18 22:55:59,457 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/77a9a75c8f8949c78f02b08e02f679fd to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/77a9a75c8f8949c78f02b08e02f679fd 2023-04-18 22:55:59,458 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f9b75e6f76574537a3e5d87abb596f46 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f9b75e6f76574537a3e5d87abb596f46 2023-04-18 22:55:59,459 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3be8684f6bb946479e2eaa7f6756716b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/3be8684f6bb946479e2eaa7f6756716b 2023-04-18 22:55:59,460 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/61a0c2e0b89a44e5b3000b09b63f9003 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/61a0c2e0b89a44e5b3000b09b63f9003 2023-04-18 22:55:59,460 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f348df132da0402b92dec3408781c94e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/f348df132da0402b92dec3408781c94e 2023-04-18 22:55:59,461 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/200d3586dd91409eb02ad18b1d79371b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/200d3586dd91409eb02ad18b1d79371b 2023-04-18 22:55:59,462 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/1bdbad68c99d4acab32bd99d01cd9af0 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/1bdbad68c99d4acab32bd99d01cd9af0 2023-04-18 22:55:59,463 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/c848574a42494feb9720bf12fd396f55 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/c848574a42494feb9720bf12fd396f55 2023-04-18 22:55:59,463 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/8bef2559405e4a8eb1d8969f4d8bb4e1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/8bef2559405e4a8eb1d8969f4d8bb4e1 2023-04-18 22:55:59,466 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/cdc0669d3a164003a6b88fa51a11bcc3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/a90d56f5256848d8b538a4be83185666, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/420d729a067d4811a47edd2279a12fc7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/524b3447f7724b88b4b15995c8fc7e52, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/1135db78873b447e825bdc0e50dccb58, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/a79eaa69d78b4bc4ba9428b2712c0edc, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/4e225cddca98488f991c2568c9346ad3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/400e68d7260f4ff9b5c44e556501fb20, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/8b47f86f8d614389ba89f4b9c7b967db, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/d1261bb3e9374f28aa0da63585b23078, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/cc2be541610e428f8d8447ac64413c15, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/b4ec40b2807048348f17a2be3d97a567, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/732e960ea8754019a2b949374ec2454e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/e43266fb897b48819693e29a5c2b55e7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/6fde858b949c439680289c6282ed425a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/fb48a873b2544956b1bf630c892243e3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/85e3b6e0f2ef449bb60b1d59692bc17d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/e1cc47c7e2784ad7a0e14f7e4612226e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/784513793b9b42888729972dbf484a12, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/d4e09678f0f049f685c353caba87d60f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/2681e280eea14814bbe2a7b4a6665e57] to archive 2023-04-18 22:55:59,466 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:55:59,467 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/cdc0669d3a164003a6b88fa51a11bcc3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/cdc0669d3a164003a6b88fa51a11bcc3 2023-04-18 22:55:59,468 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/a90d56f5256848d8b538a4be83185666 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/a90d56f5256848d8b538a4be83185666 2023-04-18 22:55:59,469 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/420d729a067d4811a47edd2279a12fc7 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/420d729a067d4811a47edd2279a12fc7 2023-04-18 22:55:59,470 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/524b3447f7724b88b4b15995c8fc7e52 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/524b3447f7724b88b4b15995c8fc7e52 2023-04-18 22:55:59,470 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/1135db78873b447e825bdc0e50dccb58 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/1135db78873b447e825bdc0e50dccb58 2023-04-18 22:55:59,471 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/a79eaa69d78b4bc4ba9428b2712c0edc to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/a79eaa69d78b4bc4ba9428b2712c0edc 2023-04-18 22:55:59,472 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/4e225cddca98488f991c2568c9346ad3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/4e225cddca98488f991c2568c9346ad3 2023-04-18 22:55:59,473 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/400e68d7260f4ff9b5c44e556501fb20 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/400e68d7260f4ff9b5c44e556501fb20 2023-04-18 22:55:59,474 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/8b47f86f8d614389ba89f4b9c7b967db to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/8b47f86f8d614389ba89f4b9c7b967db 2023-04-18 22:55:59,474 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/d1261bb3e9374f28aa0da63585b23078 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/d1261bb3e9374f28aa0da63585b23078 2023-04-18 22:55:59,475 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/cc2be541610e428f8d8447ac64413c15 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/cc2be541610e428f8d8447ac64413c15 2023-04-18 22:55:59,476 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/b4ec40b2807048348f17a2be3d97a567 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/b4ec40b2807048348f17a2be3d97a567 2023-04-18 22:55:59,477 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/732e960ea8754019a2b949374ec2454e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/732e960ea8754019a2b949374ec2454e 2023-04-18 22:55:59,478 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/e43266fb897b48819693e29a5c2b55e7 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/e43266fb897b48819693e29a5c2b55e7 2023-04-18 22:55:59,478 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/6fde858b949c439680289c6282ed425a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/6fde858b949c439680289c6282ed425a 2023-04-18 22:55:59,479 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/fb48a873b2544956b1bf630c892243e3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/fb48a873b2544956b1bf630c892243e3 2023-04-18 22:55:59,480 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/85e3b6e0f2ef449bb60b1d59692bc17d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/85e3b6e0f2ef449bb60b1d59692bc17d 2023-04-18 22:55:59,481 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/e1cc47c7e2784ad7a0e14f7e4612226e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/e1cc47c7e2784ad7a0e14f7e4612226e 2023-04-18 22:55:59,482 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/784513793b9b42888729972dbf484a12 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/784513793b9b42888729972dbf484a12 2023-04-18 22:55:59,483 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/d4e09678f0f049f685c353caba87d60f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/d4e09678f0f049f685c353caba87d60f 2023-04-18 22:55:59,484 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/2681e280eea14814bbe2a7b4a6665e57 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/2681e280eea14814bbe2a7b4a6665e57 2023-04-18 22:55:59,486 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/f71cc36de0fd47589d9a46e7d516ea06, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ccae41e792bd47329f096755ae4942d2, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/a44bde606d2245728f37834be2e9ba68, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/e9f3e612e283448a8b75f811e0c04cdb, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/b2b5fa0a20634d4198d9a90c64480a8c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/84bf13f3eae24ccda374ce4fb529af81, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/c9ada5d185f44da088c6e9f9cbd8a163, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/d4d9bf9c066f4ae4b18034c1d82758c3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/540828737208482d9582b9f440c960b3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/dd72b0e248204264a910e6bb5ab4db21, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/e5dfc00a2dbe40d5b36bcd25a3ec2304, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/9e329bc036b04f68b9055c32c5d49cd2, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/2af1eea84d9f46668b6f97970bfefa7f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/73239a68d87143beae56054ace9ad420, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/daa72ca4122f417ca9020be108e6ee35, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/c6cfee8a95e3472fb3632864aa01b629, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/2438fef9ecae42479014585888985e45, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/288cff72243a468bba66334ae25a9d5d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/9709edbacedf4a2099565e2acdec7eb3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/18c0dc233c094c43b7cf8d4b712cee0d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/551c2aecbd824e199add12c6a0b64b4d] to archive 2023-04-18 22:55:59,486 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:55:59,487 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/f71cc36de0fd47589d9a46e7d516ea06 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/f71cc36de0fd47589d9a46e7d516ea06 2023-04-18 22:55:59,488 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ccae41e792bd47329f096755ae4942d2 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/ccae41e792bd47329f096755ae4942d2 2023-04-18 22:55:59,489 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/a44bde606d2245728f37834be2e9ba68 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/a44bde606d2245728f37834be2e9ba68 2023-04-18 22:55:59,490 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/e9f3e612e283448a8b75f811e0c04cdb to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/e9f3e612e283448a8b75f811e0c04cdb 2023-04-18 22:55:59,491 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/b2b5fa0a20634d4198d9a90c64480a8c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/b2b5fa0a20634d4198d9a90c64480a8c 2023-04-18 22:55:59,491 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/84bf13f3eae24ccda374ce4fb529af81 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/84bf13f3eae24ccda374ce4fb529af81 2023-04-18 22:55:59,493 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/c9ada5d185f44da088c6e9f9cbd8a163 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/c9ada5d185f44da088c6e9f9cbd8a163 2023-04-18 22:55:59,493 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/d4d9bf9c066f4ae4b18034c1d82758c3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/d4d9bf9c066f4ae4b18034c1d82758c3 2023-04-18 22:55:59,494 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/540828737208482d9582b9f440c960b3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/540828737208482d9582b9f440c960b3 2023-04-18 22:55:59,495 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/dd72b0e248204264a910e6bb5ab4db21 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/dd72b0e248204264a910e6bb5ab4db21 2023-04-18 22:55:59,496 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/e5dfc00a2dbe40d5b36bcd25a3ec2304 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/e5dfc00a2dbe40d5b36bcd25a3ec2304 2023-04-18 22:55:59,497 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/9e329bc036b04f68b9055c32c5d49cd2 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/9e329bc036b04f68b9055c32c5d49cd2 2023-04-18 22:55:59,498 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/2af1eea84d9f46668b6f97970bfefa7f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/2af1eea84d9f46668b6f97970bfefa7f 2023-04-18 22:55:59,499 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/73239a68d87143beae56054ace9ad420 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/73239a68d87143beae56054ace9ad420 2023-04-18 22:55:59,499 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/daa72ca4122f417ca9020be108e6ee35 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/daa72ca4122f417ca9020be108e6ee35 2023-04-18 22:55:59,500 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/c6cfee8a95e3472fb3632864aa01b629 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/c6cfee8a95e3472fb3632864aa01b629 2023-04-18 22:55:59,501 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/2438fef9ecae42479014585888985e45 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/2438fef9ecae42479014585888985e45 2023-04-18 22:55:59,502 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/288cff72243a468bba66334ae25a9d5d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/288cff72243a468bba66334ae25a9d5d 2023-04-18 22:55:59,502 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/9709edbacedf4a2099565e2acdec7eb3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/9709edbacedf4a2099565e2acdec7eb3 2023-04-18 22:55:59,503 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/18c0dc233c094c43b7cf8d4b712cee0d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/18c0dc233c094c43b7cf8d4b712cee0d 2023-04-18 22:55:59,504 DEBUG [StoreCloser-TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/551c2aecbd824e199add12c6a0b64b4d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/551c2aecbd824e199add12c6a0b64b4d 2023-04-18 22:55:59,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/recovered.edits/678.seqid, newMaxSeqId=678, maxSeqId=1 2023-04-18 22:55:59,509 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0. 2023-04-18 22:55:59,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 71e1d4dea1ebcdcdd7da8f1e7108edb0: 2023-04-18 22:55:59,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:55:59,510 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=71e1d4dea1ebcdcdd7da8f1e7108edb0, regionState=CLOSED 2023-04-18 22:55:59,510 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858559510"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681858559510"}]},"ts":"1681858559510"} 2023-04-18 22:55:59,512 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-18 22:55:59,512 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 71e1d4dea1ebcdcdd7da8f1e7108edb0, server=jenkins-hbase4.apache.org,35971,1681858424869 in 1.4910 sec 2023-04-18 22:55:59,513 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-18 22:55:59,513 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=71e1d4dea1ebcdcdd7da8f1e7108edb0, UNASSIGN in 1.4950 sec 2023-04-18 22:55:59,514 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858559514"}]},"ts":"1681858559514"} 2023-04-18 22:55:59,515 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-18 22:55:59,516 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-18 22:55:59,517 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5070 sec 2023-04-18 22:56:00,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-18 22:56:00,118 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-18 22:56:00,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-18 22:56:00,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:56:00,121 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:56:00,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-18 22:56:00,122 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:56:00,126 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:56:00,128 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/recovered.edits] 2023-04-18 22:56:00,130 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/2994dd6cfbb24304a2418cb21f734340 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/2994dd6cfbb24304a2418cb21f734340 2023-04-18 22:56:00,131 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/a146a11cb883432e8393b062119d9518 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/A/a146a11cb883432e8393b062119d9518 2023-04-18 22:56:00,133 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/771431d713d1439490987ba29f8c1121 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/771431d713d1439490987ba29f8c1121 2023-04-18 22:56:00,135 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/c7b122cec2554fb3a6ff01ac7aa8a200 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/B/c7b122cec2554fb3a6ff01ac7aa8a200 2023-04-18 22:56:00,136 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/5d91cb8fe77b43549dda2cbbd0b1d5a4 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/5d91cb8fe77b43549dda2cbbd0b1d5a4 2023-04-18 22:56:00,137 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/b311e211aeda468eb0e42e0fea23aecd to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/C/b311e211aeda468eb0e42e0fea23aecd 2023-04-18 22:56:00,140 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/recovered.edits/678.seqid to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0/recovered.edits/678.seqid 2023-04-18 22:56:00,140 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/71e1d4dea1ebcdcdd7da8f1e7108edb0 2023-04-18 22:56:00,140 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 22:56:00,142 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:56:00,144 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-18 22:56:00,145 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-18 22:56:00,146 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:56:00,146 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-18 22:56:00,146 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681858560146"}]},"ts":"9223372036854775807"} 2023-04-18 22:56:00,148 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-18 22:56:00,148 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 71e1d4dea1ebcdcdd7da8f1e7108edb0, NAME => 'TestAcidGuarantees,,1681858535193.71e1d4dea1ebcdcdd7da8f1e7108edb0.', STARTKEY => '', ENDKEY => ''}] 2023-04-18 22:56:00,148 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-18 22:56:00,148 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681858560148"}]},"ts":"9223372036854775807"} 2023-04-18 22:56:00,149 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-18 22:56:00,151 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:56:00,152 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 32 msec 2023-04-18 22:56:00,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-18 22:56:00,223 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-18 22:56:00,234 INFO [Listener at localhost/36651] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229 (was 227) - Thread LEAK? -, OpenFileDescriptor=575 (was 521) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=383 (was 373) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=6606 (was 6201) - AvailableMemoryMB LEAK? - 2023-04-18 22:56:00,247 INFO [Listener at localhost/36651] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229, OpenFileDescriptor=575, MaxFileDescriptor=60000, SystemLoadAverage=383, ProcessCount=169, AvailableMemoryMB=6605 2023-04-18 22:56:00,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 22:56:00,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 22:56:00,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-18 22:56:00,251 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-18 22:56:00,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-18 22:56:00,252 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-18 22:56:00,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-18 22:56:00,253 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:00,253 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78 empty. 2023-04-18 22:56:00,253 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:00,253 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 22:56:00,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-18 22:56:00,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-18 22:56:00,660 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 22:56:00,662 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8181c42622934944fa97cf06c7004a78, NAME => 'TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp 2023-04-18 22:56:00,667 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:56:00,667 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 8181c42622934944fa97cf06c7004a78, disabling compactions & flushes 2023-04-18 22:56:00,667 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:00,667 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:00,667 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. after waiting 0 ms 2023-04-18 22:56:00,667 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:00,667 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:00,667 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:00,669 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-18 22:56:00,670 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858560670"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681858560670"}]},"ts":"1681858560670"} 2023-04-18 22:56:00,671 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-18 22:56:00,672 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-18 22:56:00,672 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858560672"}]},"ts":"1681858560672"} 2023-04-18 22:56:00,673 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-18 22:56:00,676 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8181c42622934944fa97cf06c7004a78, ASSIGN}] 2023-04-18 22:56:00,678 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8181c42622934944fa97cf06c7004a78, ASSIGN 2023-04-18 22:56:00,678 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8181c42622934944fa97cf06c7004a78, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35971,1681858424869; forceNewPlan=false, retain=false 2023-04-18 22:56:00,829 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=8181c42622934944fa97cf06c7004a78, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:00,829 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858560829"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858560829"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858560829"}]},"ts":"1681858560829"} 2023-04-18 22:56:00,831 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:56:00,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-18 22:56:00,985 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:00,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8181c42622934944fa97cf06c7004a78, NAME => 'TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.', STARTKEY => '', ENDKEY => ''} 2023-04-18 22:56:00,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:00,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:56:00,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:00,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:00,987 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:00,988 DEBUG [StoreOpener-8181c42622934944fa97cf06c7004a78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A 2023-04-18 22:56:00,988 DEBUG [StoreOpener-8181c42622934944fa97cf06c7004a78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A 2023-04-18 22:56:00,988 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:56:00,988 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8181c42622934944fa97cf06c7004a78 columnFamilyName A 2023-04-18 22:56:00,989 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.HStore(310): Store=8181c42622934944fa97cf06c7004a78/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:56:00,989 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:00,990 DEBUG [StoreOpener-8181c42622934944fa97cf06c7004a78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B 2023-04-18 22:56:00,990 DEBUG [StoreOpener-8181c42622934944fa97cf06c7004a78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B 2023-04-18 22:56:00,990 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:56:00,990 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8181c42622934944fa97cf06c7004a78 columnFamilyName B 2023-04-18 22:56:00,991 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.HStore(310): Store=8181c42622934944fa97cf06c7004a78/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:56:00,991 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:00,992 DEBUG [StoreOpener-8181c42622934944fa97cf06c7004a78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C 2023-04-18 22:56:00,992 DEBUG [StoreOpener-8181c42622934944fa97cf06c7004a78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C 2023-04-18 22:56:00,992 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:56:00,992 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8181c42622934944fa97cf06c7004a78 columnFamilyName C 2023-04-18 22:56:00,993 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.HStore(310): Store=8181c42622934944fa97cf06c7004a78/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:56:00,993 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:00,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:00,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:00,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 22:56:00,996 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:00,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-18 22:56:00,998 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8181c42622934944fa97cf06c7004a78; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61150805, jitterRate=-0.08878199756145477}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 22:56:00,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:00,999 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., pid=56, masterSystemTime=1681858560982 2023-04-18 22:56:01,000 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:01,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:01,000 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=8181c42622934944fa97cf06c7004a78, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:01,000 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858561000"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681858561000"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681858561000"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681858561000"}]},"ts":"1681858561000"} 2023-04-18 22:56:01,002 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-18 22:56:01,002 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 in 170 msec 2023-04-18 22:56:01,003 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-18 22:56:01,003 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8181c42622934944fa97cf06c7004a78, ASSIGN in 326 msec 2023-04-18 22:56:01,004 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-18 22:56:01,004 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858561004"}]},"ts":"1681858561004"} 2023-04-18 22:56:01,005 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-18 22:56:01,007 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-18 22:56:01,008 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 758 msec 2023-04-18 22:56:01,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-18 22:56:01,355 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-18 22:56:01,356 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x1e3503b4 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:56:01,361 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63da10ca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:56:01,362 DEBUG [Listener at localhost/36651] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:56:01,364 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45786, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:56:01,365 DEBUG [Listener at localhost/36651] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-18 22:56:01,367 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47118, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-18 22:56:01,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-18 22:56:01,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-18 22:56:01,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-18 22:56:01,782 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-18 22:56:01,783 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-18 22:56:01,783 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-18 22:56:01,785 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-18 22:56:01,788 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8181c42622934944fa97cf06c7004a78, REOPEN/MOVE}] 2023-04-18 22:56:01,788 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8181c42622934944fa97cf06c7004a78, REOPEN/MOVE 2023-04-18 22:56:01,789 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=8181c42622934944fa97cf06c7004a78, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:01,789 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858561789"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858561789"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858561789"}]},"ts":"1681858561789"} 2023-04-18 22:56:01,790 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:56:01,941 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:01,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8181c42622934944fa97cf06c7004a78, disabling compactions & flushes 2023-04-18 22:56:01,942 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:01,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:01,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. after waiting 0 ms 2023-04-18 22:56:01,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:01,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-18 22:56:01,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:01,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:01,946 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 8181c42622934944fa97cf06c7004a78 to self. 2023-04-18 22:56:01,947 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:01,948 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=8181c42622934944fa97cf06c7004a78, regionState=CLOSED 2023-04-18 22:56:01,948 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858561948"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681858561948"}]},"ts":"1681858561948"} 2023-04-18 22:56:01,950 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-18 22:56:01,950 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 in 159 msec 2023-04-18 22:56:01,950 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8181c42622934944fa97cf06c7004a78, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35971,1681858424869; forceNewPlan=false, retain=true 2023-04-18 22:56:02,101 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=8181c42622934944fa97cf06c7004a78, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,101 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858562101"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858562101"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858562101"}]},"ts":"1681858562101"} 2023-04-18 22:56:02,102 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:56:02,257 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:02,257 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8181c42622934944fa97cf06c7004a78, NAME => 'TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.', STARTKEY => '', ENDKEY => ''} 2023-04-18 22:56:02,257 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:02,257 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-18 22:56:02,257 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:02,257 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:02,258 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:02,259 DEBUG [StoreOpener-8181c42622934944fa97cf06c7004a78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A 2023-04-18 22:56:02,259 DEBUG [StoreOpener-8181c42622934944fa97cf06c7004a78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A 2023-04-18 22:56:02,259 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:56:02,259 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8181c42622934944fa97cf06c7004a78 columnFamilyName A 2023-04-18 22:56:02,261 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.HStore(310): Store=8181c42622934944fa97cf06c7004a78/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:56:02,261 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:02,262 DEBUG [StoreOpener-8181c42622934944fa97cf06c7004a78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B 2023-04-18 22:56:02,262 DEBUG [StoreOpener-8181c42622934944fa97cf06c7004a78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B 2023-04-18 22:56:02,262 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:56:02,262 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8181c42622934944fa97cf06c7004a78 columnFamilyName B 2023-04-18 22:56:02,262 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.HStore(310): Store=8181c42622934944fa97cf06c7004a78/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:56:02,262 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:02,263 DEBUG [StoreOpener-8181c42622934944fa97cf06c7004a78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C 2023-04-18 22:56:02,263 DEBUG [StoreOpener-8181c42622934944fa97cf06c7004a78-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C 2023-04-18 22:56:02,263 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-18 22:56:02,263 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8181c42622934944fa97cf06c7004a78 columnFamilyName C 2023-04-18 22:56:02,264 INFO [StoreOpener-8181c42622934944fa97cf06c7004a78-1] regionserver.HStore(310): Store=8181c42622934944fa97cf06c7004a78/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-18 22:56:02,264 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:02,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:02,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:02,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-18 22:56:02,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:02,269 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8181c42622934944fa97cf06c7004a78; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62149192, jitterRate=-0.07390487194061279}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-18 22:56:02,269 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:02,269 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., pid=61, masterSystemTime=1681858562254 2023-04-18 22:56:02,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:02,270 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:02,271 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=8181c42622934944fa97cf06c7004a78, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,271 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858562271"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681858562271"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681858562271"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681858562271"}]},"ts":"1681858562271"} 2023-04-18 22:56:02,273 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-18 22:56:02,273 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 in 170 msec 2023-04-18 22:56:02,274 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-18 22:56:02,274 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8181c42622934944fa97cf06c7004a78, REOPEN/MOVE in 485 msec 2023-04-18 22:56:02,276 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-18 22:56:02,276 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 489 msec 2023-04-18 22:56:02,277 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 907 msec 2023-04-18 22:56:02,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-18 22:56:02,279 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x6f06dd50 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:56:02,286 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59185ba7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:56:02,286 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e8654 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:56:02,293 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62b3caa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:56:02,293 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x4db22237 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:56:02,306 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66b5b1f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:56:02,307 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x08b57aae to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:56:02,312 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@544a4215, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:56:02,312 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x06799455 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:56:02,320 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25baee3e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:56:02,320 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x2c53bf96 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:56:02,331 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e6a6f3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:56:02,331 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x3b59bdcf to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:56:02,339 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a470a3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:56:02,339 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x3d765193 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:56:02,346 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bb72437, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:56:02,347 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x32bf16cd to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:56:02,356 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3dca23c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:56:02,356 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(139): Connect 0x28164190 to 127.0.0.1:50825 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-18 22:56:02,361 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62d52321, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-18 22:56:02,374 DEBUG [hconnection-0x3877f414-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:56:02,374 DEBUG [hconnection-0x4927eb99-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:56:02,375 DEBUG [hconnection-0x545952c7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:56:02,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:02,376 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45830, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:56:02,380 DEBUG [hconnection-0x5140efdc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:56:02,385 DEBUG [hconnection-0x5375fba8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:56:02,385 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45800, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:56:02,385 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45814, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:56:02,385 DEBUG [hconnection-0x75a7abd6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:56:02,385 DEBUG [hconnection-0x1da2cd8b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:56:02,389 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45836, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:56:02,391 DEBUG [hconnection-0x331af02b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:56:02,391 DEBUG [hconnection-0xfef51db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:56:02,391 DEBUG [hconnection-0x4d06e8f3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-18 22:56:02,392 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45846, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:56:02,393 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45876, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:56:02,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:02,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:02,394 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:02,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:02,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:02,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:02,396 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:02,397 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:02,397 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45844, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:56:02,398 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45852, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:56:02,398 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45860, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:56:02,398 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45890, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-18 22:56:02,402 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:02,404 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:02,404 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,404 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:02,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:02,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:02,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:02,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:02,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:02,408 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:02,408 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:02,408 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:02,409 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:02,409 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:02,409 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:02,409 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:02,410 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-18 22:56:02,411 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:02,411 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:02,411 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:02,411 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:02,411 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:02,411 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:02,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:02,413 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:02,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858562443, max timestamp=1681858562449Num uniques 859; 2023-04-18 22:56:02,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:56:02,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858562443, max timestamp=1681858562449Num uniques 859; 2023-04-18 22:56:02,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:56:02,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858562443, max timestamp=1681858562449Num uniques 859; 2023-04-18 22:56:02,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:56:02,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7681, totalHeapSize=17712, min timestamp=1681858562461, max timestamp=1681858562468Num uniques 1018; 2023-04-18 22:56:02,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46611, new segment data size=7681, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-18 22:56:02,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858562461, max timestamp=1681858562468Num uniques 1018; 2023-04-18 22:56:02,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-18 22:56:02,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858562461, max timestamp=1681858562468Num uniques 1018; 2023-04-18 22:56:02,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-18 22:56:02,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:02,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45860 deadline: 1681858622488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12677, totalHeapSize=29048, min timestamp=1681858562478, max timestamp=1681858562490Num uniques 1027; 2023-04-18 22:56:02,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47027, new segment data size=12677, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-18 22:56:02,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681858562478, max timestamp=1681858562489Num uniques 1027; 2023-04-18 22:56:02,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-18 22:56:02,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681858562478, max timestamp=1681858562489Num uniques 1027; 2023-04-18 22:56:02,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-18 22:56:02,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:02,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858622505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:02,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858622506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:02,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858622507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:02,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858622508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:02,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45860 deadline: 1681858622594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:02,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858622609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:02,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858622609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:02,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858622609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:02,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858622611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:02,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45860 deadline: 1681858622797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:02,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858622812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:02,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858622813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:02,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858622813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:02,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858622813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:02,832 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041834e6ef0b752b47549dac402f4a9ef87d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041834e6ef0b752b47549dac402f4a9ef87d 2023-04-18 22:56:03,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:03,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45860 deadline: 1681858623100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:03,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858623115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:03,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:03,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858623115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858623115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:03,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858623117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,237 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=13, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/610f19eb126d49fcbfe93b92359eaaf9 2023-04-18 22:56:03,247 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/db372612858246aaa4e907e6c9179558 2023-04-18 22:56:03,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:03,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45860 deadline: 1681858623604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:03,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858623618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:03,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858623619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:03,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858623619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:03,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858623625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,659 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/81fcb5c4ae9442dfb00c2c696f9fcf43 2023-04-18 22:56:03,663 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/610f19eb126d49fcbfe93b92359eaaf9 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/610f19eb126d49fcbfe93b92359eaaf9 2023-04-18 22:56:03,667 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/610f19eb126d49fcbfe93b92359eaaf9, entries=100, sequenceid=13, filesize=18.4 K 2023-04-18 22:56:03,668 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/db372612858246aaa4e907e6c9179558 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/db372612858246aaa4e907e6c9179558 2023-04-18 22:56:03,672 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/db372612858246aaa4e907e6c9179558, entries=100, sequenceid=13, filesize=9.3 K 2023-04-18 22:56:03,673 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/81fcb5c4ae9442dfb00c2c696f9fcf43 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/81fcb5c4ae9442dfb00c2c696f9fcf43 2023-04-18 22:56:03,676 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/81fcb5c4ae9442dfb00c2c696f9fcf43, entries=100, sequenceid=13, filesize=9.3 K 2023-04-18 22:56:03,677 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 8181c42622934944fa97cf06c7004a78 in 1267ms, sequenceid=13, compaction requested=false 2023-04-18 22:56:03,677 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:03,677 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:03,677 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:03,678 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:03,678 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,678 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:03,678 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:03,679 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:03,679 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:03,680 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:03,680 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:03,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:03,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a3c5090[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:03,682 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:03,682 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:03,683 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:03,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:03,683 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:03,683 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:03,683 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,684 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:03,683 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:03,685 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:03,685 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,685 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:03,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:03,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:03,685 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:03,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:03,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:03,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:03,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:03,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:03,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:03,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:03,688 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:03,688 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:03,688 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:03,688 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:03,688 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:03,690 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:03,690 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:03,690 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:03,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:03,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:03,691 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:03,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:03,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,692 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:03,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:03,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,695 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:03,695 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:03,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:03,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:03,695 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,695 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:03,695 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:03,696 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:03,695 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:03,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:03,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:03,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:03,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:03,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:03,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:03,796 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:03,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:03,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:03,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:03,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:03,799 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:03,799 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:03,799 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:03,799 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:03,800 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:03,800 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:03,802 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,802 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:03,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:03,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:03,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:03,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:03,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:03,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:03,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:03,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:03,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:03,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:03,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:03,804 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:03,804 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:03,804 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:03,804 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=140.89 KB heapSize=393.84 KB 2023-04-18 22:56:03,805 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:03,805 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:03,805 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:03,805 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:03,805 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:03,805 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:04,222 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304182889c453ac6e4cff96fd8fa3643d2959 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304182889c453ac6e4cff96fd8fa3643d2959 2023-04-18 22:56:04,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:04,612 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:04,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:04,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858624624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:04,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858624626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,628 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/baffd83b791a40048c07bf232167243d 2023-04-18 22:56:04,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:04,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45860 deadline: 1681858624627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:04,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858624629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:04,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858624632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,662 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/1fa99d3246484e90ad97e46a02fe5ba1 2023-04-18 22:56:04,679 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/1c6a68bd4348419d87ece43fec1e4ee6 2023-04-18 22:56:04,684 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/baffd83b791a40048c07bf232167243d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/baffd83b791a40048c07bf232167243d 2023-04-18 22:56:04,688 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/baffd83b791a40048c07bf232167243d, entries=150, sequenceid=83, filesize=25.1 K 2023-04-18 22:56:04,689 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/1fa99d3246484e90ad97e46a02fe5ba1 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/1fa99d3246484e90ad97e46a02fe5ba1 2023-04-18 22:56:04,693 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/1fa99d3246484e90ad97e46a02fe5ba1, entries=150, sequenceid=83, filesize=11.6 K 2023-04-18 22:56:04,694 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/1c6a68bd4348419d87ece43fec1e4ee6 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/1c6a68bd4348419d87ece43fec1e4ee6 2023-04-18 22:56:04,697 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/1c6a68bd4348419d87ece43fec1e4ee6, entries=150, sequenceid=83, filesize=11.6 K 2023-04-18 22:56:04,698 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~393.80 KB/403248, currentSize=40.25 KB/41220 for 8181c42622934944fa97cf06c7004a78 in 894ms, sequenceid=83, compaction requested=false 2023-04-18 22:56:04,698 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:04,698 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:04,698 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:04,698 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:04,698 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,698 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:04,698 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:04,700 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:04,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:04,700 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:04,700 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,701 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:04,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:04,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:04,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:04,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@301be585[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:04,702 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:04,702 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,704 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,705 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:04,705 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:04,705 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:04,705 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,705 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:04,706 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:04,706 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:04,706 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:04,706 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:04,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:04,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:04,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:04,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:04,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:04,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:04,709 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:04,709 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:04,709 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:04,709 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:04,709 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:04,711 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,711 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:04,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:04,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,711 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:04,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:04,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:04,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:04,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,713 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:04,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,714 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:04,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,717 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:04,717 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:04,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:04,717 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,717 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:04,717 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:04,718 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,718 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,718 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,718 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,718 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:04,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:04,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:04,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:04,719 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:04,719 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:04,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:04,733 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:56:04,733 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:04,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:04,733 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:04,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:04,733 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:04,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:04,753 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041865c0da05276b42419cd6286147f2ae6e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041865c0da05276b42419cd6286147f2ae6e 2023-04-18 22:56:04,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858564778, max timestamp=1681858564803Num uniques 859; 2023-04-18 22:56:04,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858564778, max timestamp=1681858564803Num uniques 859; 2023-04-18 22:56:04,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858564778, max timestamp=1681858564803Num uniques 859; 2023-04-18 22:56:04,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:56:04,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:56:04,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:56:04,819 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:04,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:04,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:04,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:04,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:04,822 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:04,822 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:04,822 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:04,822 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:04,823 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,823 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:04,829 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,829 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:04,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:04,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:04,830 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,830 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:04,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:04,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:04,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:04,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:04,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:04,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:04,831 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:04,831 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:04,831 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:04,836 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:04,836 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:04,836 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:04,837 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:04,837 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:04,837 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:04,837 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:04,837 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:04,837 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,837 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:04,837 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:04,844 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,844 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:04,844 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,844 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:04,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:04,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ded5bb6[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:04,846 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:04,847 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,848 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,848 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:04,848 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:04,848 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:04,849 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,849 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:04,852 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:04,852 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,852 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:04,852 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:04,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:04,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:04,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:04,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:04,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:04,857 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:04,857 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:04,857 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:04,857 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:04,857 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:04,859 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,859 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:04,859 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:04,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:04,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,860 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,861 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:04,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,864 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,864 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:04,864 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:04,864 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,864 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,864 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,864 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:04,864 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:04,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:04,864 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:04,865 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:04,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:04,865 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:04,865 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:04,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:04,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:04,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:04,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45860 deadline: 1681858624864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,965 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:04,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:04,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:04,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:04,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:04,969 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:04,969 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:04,969 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:04,969 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:04,969 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,969 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:04,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:04,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45860 deadline: 1681858624969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,971 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,971 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:04,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:04,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:04,971 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,971 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:04,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:04,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:04,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:04,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:04,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:04,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:04,972 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:04,973 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:04,973 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:04,973 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:04,973 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:04,973 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:04,973 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:04,973 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:04,973 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:04,973 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,973 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:04,973 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:04,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,976 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:04,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,976 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:04,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:04,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:04,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2410cb4e[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:04,978 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:04,978 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,979 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,979 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:04,979 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:04,979 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:04,979 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,979 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:04,981 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,981 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:04,981 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:04,981 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:04,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:04,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:04,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:04,984 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:04,984 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:04,984 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:04,984 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:04,984 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:04,985 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,985 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:04,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:04,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,985 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:04,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:04,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:04,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:04,986 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,987 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:04,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,991 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,991 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:04,991 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:04,991 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:04,991 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:04,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:04,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:04,991 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:04,991 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:04,992 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:04,992 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:04,992 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:04,992 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:04,992 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:04,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:05,033 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-18 22:56:05,092 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:05,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:05,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:05,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:05,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:05,096 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:05,096 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:05,096 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:05,096 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:05,096 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,096 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:05,099 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,099 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:05,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:05,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:05,099 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,099 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:05,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:05,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:05,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:05,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:05,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:05,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:05,101 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:05,101 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:05,101 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:05,104 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:05,104 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:05,104 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:05,104 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:05,104 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:05,104 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:05,104 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:05,104 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:05,104 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,104 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:05,104 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:05,107 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,107 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:05,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:05,107 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,107 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:05,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:05,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:05,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:05,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:05,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:05,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c80b78c[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:05,109 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:05,109 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,111 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:05,112 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:05,112 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,112 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:05,115 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,115 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:05,115 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:05,115 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:05,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:05,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:05,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:05,118 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:05,118 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:05,118 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:05,118 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:05,118 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:05,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:05,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:05,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:05,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:05,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:05,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:05,122 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:05,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:05,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:05,123 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:05,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:05,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,126 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,126 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:05,126 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:05,126 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:05,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:05,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:05,126 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:05,126 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:05,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:05,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:05,126 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:05,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:05,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:05,127 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,127 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,127 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,127 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:05,127 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:05,167 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=93, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/e4a809f9509b42a3bfaa677798307ce1 2023-04-18 22:56:05,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:05,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45860 deadline: 1681858625172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,185 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/73c14ff73b4d4855be9efc912cdf8eb7 2023-04-18 22:56:05,210 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/56397fbac6c2440caf04ba166764019d 2023-04-18 22:56:05,215 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/e4a809f9509b42a3bfaa677798307ce1 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e4a809f9509b42a3bfaa677798307ce1 2023-04-18 22:56:05,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e4a809f9509b42a3bfaa677798307ce1, entries=200, sequenceid=93, filesize=31.9 K 2023-04-18 22:56:05,224 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/73c14ff73b4d4855be9efc912cdf8eb7 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/73c14ff73b4d4855be9efc912cdf8eb7 2023-04-18 22:56:05,227 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:05,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:05,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/73c14ff73b4d4855be9efc912cdf8eb7, entries=150, sequenceid=93, filesize=11.6 K 2023-04-18 22:56:05,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:05,230 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/56397fbac6c2440caf04ba166764019d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/56397fbac6c2440caf04ba166764019d 2023-04-18 22:56:05,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:05,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:05,231 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:05,231 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:05,231 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:05,231 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:05,231 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,231 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:05,233 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:05,233 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:05,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:05,234 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/56397fbac6c2440caf04ba166764019d, entries=150, sequenceid=93, filesize=11.6 K 2023-04-18 22:56:05,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:05,234 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,234 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:05,235 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 8181c42622934944fa97cf06c7004a78 in 502ms, sequenceid=93, compaction requested=true 2023-04-18 22:56:05,235 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:05,235 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:05,235 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:56:05,235 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:56:05,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,235 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:05,235 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:05,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:05,236 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77230 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:56:05,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:05,236 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/A is initiating minor compaction (all files) 2023-04-18 22:56:05,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:05,236 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:56:05,236 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:05,236 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:05,236 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/A in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:05,236 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:05,236 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/B is initiating minor compaction (all files) 2023-04-18 22:56:05,236 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/610f19eb126d49fcbfe93b92359eaaf9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/baffd83b791a40048c07bf232167243d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e4a809f9509b42a3bfaa677798307ce1] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=75.4 K 2023-04-18 22:56:05,236 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/B in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:05,236 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:05,236 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/db372612858246aaa4e907e6c9179558, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/1fa99d3246484e90ad97e46a02fe5ba1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/73c14ff73b4d4855be9efc912cdf8eb7] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=32.5 K 2023-04-18 22:56:05,236 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:05,237 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:05,237 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:05,237 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-18 22:56:05,237 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 610f19eb126d49fcbfe93b92359eaaf9, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681858562408 2023-04-18 22:56:05,237 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting db372612858246aaa4e907e6c9179558, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681858562408 2023-04-18 22:56:05,237 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:05,237 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:05,237 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:05,237 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:05,237 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting baffd83b791a40048c07bf232167243d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681858562505 2023-04-18 22:56:05,237 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 1fa99d3246484e90ad97e46a02fe5ba1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681858562505 2023-04-18 22:56:05,238 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:05,238 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:05,238 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting e4a809f9509b42a3bfaa677798307ce1, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681858564617 2023-04-18 22:56:05,239 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 73c14ff73b4d4855be9efc912cdf8eb7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681858564623 2023-04-18 22:56:05,261 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#B#compaction#497 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:05,269 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#A#compaction#496 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:05,312 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418195ff78e320949d49f9c4bcff81b3a59 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418195ff78e320949d49f9c4bcff81b3a59 2023-04-18 22:56:05,317 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=130, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/564279ea5ab743579c69e4f7f920680f 2023-04-18 22:56:05,337 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/4caccb48ebc340caa21b3d8f946ca0b0 2023-04-18 22:56:05,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:05,478 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:05,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:05,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45860 deadline: 1681858625498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:05,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45860 deadline: 1681858625602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,692 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/070ea5ba645745b4b02cd08c2efba7cd as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/070ea5ba645745b4b02cd08c2efba7cd 2023-04-18 22:56:05,696 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8181c42622934944fa97cf06c7004a78/B of 8181c42622934944fa97cf06c7004a78 into 070ea5ba645745b4b02cd08c2efba7cd(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:05,696 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:05,697 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/B, priority=13, startTime=1681858565235; duration=0sec 2023-04-18 22:56:05,697 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:05,697 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:56:05,698 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:56:05,698 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/C is initiating minor compaction (all files) 2023-04-18 22:56:05,698 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/C in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:05,698 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/81fcb5c4ae9442dfb00c2c696f9fcf43, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/1c6a68bd4348419d87ece43fec1e4ee6, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/56397fbac6c2440caf04ba166764019d] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=32.5 K 2023-04-18 22:56:05,698 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 81fcb5c4ae9442dfb00c2c696f9fcf43, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681858562408 2023-04-18 22:56:05,699 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 1c6a68bd4348419d87ece43fec1e4ee6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681858562505 2023-04-18 22:56:05,699 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 56397fbac6c2440caf04ba166764019d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681858564623 2023-04-18 22:56:05,706 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#C#compaction#501 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:05,723 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/54826ebf3eae4c38b6e32fb81866e3ef as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/54826ebf3eae4c38b6e32fb81866e3ef 2023-04-18 22:56:05,726 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8181c42622934944fa97cf06c7004a78/C of 8181c42622934944fa97cf06c7004a78 into 54826ebf3eae4c38b6e32fb81866e3ef(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:05,726 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:05,726 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/C, priority=13, startTime=1681858565235; duration=0sec 2023-04-18 22:56:05,727 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:05,752 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/106c7305baeb440283898321451e943a 2023-04-18 22:56:05,756 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/564279ea5ab743579c69e4f7f920680f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/564279ea5ab743579c69e4f7f920680f 2023-04-18 22:56:05,761 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/564279ea5ab743579c69e4f7f920680f, entries=150, sequenceid=130, filesize=25.2 K 2023-04-18 22:56:05,762 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/4caccb48ebc340caa21b3d8f946ca0b0 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/4caccb48ebc340caa21b3d8f946ca0b0 2023-04-18 22:56:05,765 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/4caccb48ebc340caa21b3d8f946ca0b0, entries=150, sequenceid=130, filesize=11.7 K 2023-04-18 22:56:05,766 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/106c7305baeb440283898321451e943a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/106c7305baeb440283898321451e943a 2023-04-18 22:56:05,770 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/106c7305baeb440283898321451e943a, entries=150, sequenceid=130, filesize=11.7 K 2023-04-18 22:56:05,771 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 8181c42622934944fa97cf06c7004a78 in 534ms, sequenceid=130, compaction requested=false 2023-04-18 22:56:05,771 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:05,771 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:05,771 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:05,771 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:05,771 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,771 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:05,771 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:05,773 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,773 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:05,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:05,774 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,774 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:05,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:05,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:05,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:05,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:05,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:05,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63585c54[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:05,776 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:05,776 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,778 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,778 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:05,778 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:05,778 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:05,778 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,778 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:05,782 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,782 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:05,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:05,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:05,782 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:05,782 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:05,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:05,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:05,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:05,785 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:05,786 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:05,786 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:05,786 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:05,786 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:05,788 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,788 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:05,788 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:05,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:05,789 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:05,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:05,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:05,790 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:05,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:05,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,793 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,793 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:05,793 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:05,793 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,793 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,793 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,793 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:05,793 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:05,793 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:05,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:05,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:05,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:05,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:05,793 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:05,794 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:05,794 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:05,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:05,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:05,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:05,807 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-18 22:56:05,808 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:05,808 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:05,808 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:05,808 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:05,808 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:05,808 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:05,832 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304187ec1633d56d64437a62b858726772a1d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187ec1633d56d64437a62b858726772a1d 2023-04-18 22:56:05,852 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=143, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/cc878e24d7f24b159221bf03392b1805 2023-04-18 22:56:05,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858565828, max timestamp=1681858565874Num uniques 859; 2023-04-18 22:56:05,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:56:05,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858565828, max timestamp=1681858565874Num uniques 859; 2023-04-18 22:56:05,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:56:05,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858565828, max timestamp=1681858565874Num uniques 859; 2023-04-18 22:56:05,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:56:05,882 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/73955fc37e194e778ef596234acc9a11 2023-04-18 22:56:05,894 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:05,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:05,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:05,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:05,899 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/3ef7f5086e1c4ec89c632e7e4513fd28 2023-04-18 22:56:05,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:05,899 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:05,899 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:05,899 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:05,899 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:05,900 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,900 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:05,902 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:05,902 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:05,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:05,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:05,903 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,903 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:05,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:05,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:05,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:05,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:05,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:05,904 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:05,905 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:05,905 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:05,905 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/cc878e24d7f24b159221bf03392b1805 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/cc878e24d7f24b159221bf03392b1805 2023-04-18 22:56:05,905 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:05,905 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:05,905 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:05,905 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:05,905 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:05,905 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:05,905 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:05,905 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:05,905 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,905 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:05,905 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:05,907 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,907 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:05,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:05,908 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,908 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:05,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:05,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:05,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/cc878e24d7f24b159221bf03392b1805, entries=150, sequenceid=143, filesize=25.3 K 2023-04-18 22:56:05,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:05,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:05,910 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/73955fc37e194e778ef596234acc9a11 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/73955fc37e194e778ef596234acc9a11 2023-04-18 22:56:05,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:05,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@646f8b62[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:05,910 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:05,910 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,912 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,912 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:05,912 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:05,912 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:05,912 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,912 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:05,913 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:05,914 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:05,914 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:05,914 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,914 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/73955fc37e194e778ef596234acc9a11, entries=150, sequenceid=143, filesize=11.8 K 2023-04-18 22:56:05,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:05,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:05,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:05,915 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/3ef7f5086e1c4ec89c632e7e4513fd28 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/3ef7f5086e1c4ec89c632e7e4513fd28 2023-04-18 22:56:05,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:05,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:05,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:05,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:05,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:05,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:05,917 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:05,917 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:05,917 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:05,917 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:05,917 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:05,918 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/3ef7f5086e1c4ec89c632e7e4513fd28, entries=150, sequenceid=143, filesize=11.8 K 2023-04-18 22:56:05,919 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=87.22 KB/89310 for 8181c42622934944fa97cf06c7004a78 in 112ms, sequenceid=143, compaction requested=true 2023-04-18 22:56:05,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:05,919 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-18 22:56:05,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:05,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:05,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-18 22:56:05,920 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-18 22:56:05,920 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-18 22:56:05,920 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. because compaction request was cancelled 2023-04-18 22:56:05,920 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:56:05,921 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35961 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:56:05,921 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/B is initiating minor compaction (all files) 2023-04-18 22:56:05,922 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/B in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:05,922 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/070ea5ba645745b4b02cd08c2efba7cd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/4caccb48ebc340caa21b3d8f946ca0b0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/73955fc37e194e778ef596234acc9a11] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=35.1 K 2023-04-18 22:56:05,922 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 070ea5ba645745b4b02cd08c2efba7cd, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681858564623 2023-04-18 22:56:05,923 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 4caccb48ebc340caa21b3d8f946ca0b0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1681858564852 2023-04-18 22:56:05,923 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 73955fc37e194e778ef596234acc9a11, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681858565490 2023-04-18 22:56:05,924 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,924 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:05,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,924 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:05,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:05,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:05,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:05,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:05,927 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:05,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:05,928 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=93.93 KB heapSize=265.45 KB 2023-04-18 22:56:05,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:05,929 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:05,929 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:05,929 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:05,929 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:05,929 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:05,929 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:05,929 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,930 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#B#compaction#505 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:05,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:05,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,933 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:05,933 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:05,933 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:05,933 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:05,933 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:05,934 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:05,934 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:05,934 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,934 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:05,934 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:05,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:05,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:05,934 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:05,934 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:05,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:05,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:05,969 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041896d17c3f159941c891f221ae6d7b7116 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041896d17c3f159941c891f221ae6d7b7116 2023-04-18 22:56:05,979 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/5364433954ce4561814e30819c239fb8 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/5364433954ce4561814e30819c239fb8 2023-04-18 22:56:05,983 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8181c42622934944fa97cf06c7004a78/B of 8181c42622934944fa97cf06c7004a78 into 5364433954ce4561814e30819c239fb8(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:05,983 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:05,983 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/B, priority=13, startTime=1681858565919; duration=0sec 2023-04-18 22:56:05,984 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:05,984 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-18 22:56:05,985 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35961 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-18 22:56:05,985 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/C is initiating minor compaction (all files) 2023-04-18 22:56:05,986 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/C in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:05,986 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/54826ebf3eae4c38b6e32fb81866e3ef, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/106c7305baeb440283898321451e943a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/3ef7f5086e1c4ec89c632e7e4513fd28] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=35.1 K 2023-04-18 22:56:05,986 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 54826ebf3eae4c38b6e32fb81866e3ef, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681858564623 2023-04-18 22:56:05,986 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 106c7305baeb440283898321451e943a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1681858564852 2023-04-18 22:56:05,987 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 3ef7f5086e1c4ec89c632e7e4513fd28, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681858565490 2023-04-18 22:56:06,009 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#C#compaction#507 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:06,010 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=174, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/d151bf2efe1d48308a7617c53b2a6029 2023-04-18 22:56:06,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45860 deadline: 1681858626007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,034 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:06,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:06,036 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/3bc6d813bc334a5f8e3215b43aafebd3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/3bc6d813bc334a5f8e3215b43aafebd3 2023-04-18 22:56:06,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:06,041 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8181c42622934944fa97cf06c7004a78/C of 8181c42622934944fa97cf06c7004a78 into 3bc6d813bc334a5f8e3215b43aafebd3(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:06,041 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:06,041 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/C, priority=13, startTime=1681858565919; duration=0sec 2023-04-18 22:56:06,041 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:06,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:06,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:06,048 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:06,048 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:06,048 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:06,048 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:06,049 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,049 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,050 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:06,050 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:06,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:06,050 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,050 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:06,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:06,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:06,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:06,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:06,051 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:06,051 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:06,051 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:06,052 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:06,052 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:06,052 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:06,052 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:06,052 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:06,052 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:06,052 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:06,052 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:06,052 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,052 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:06,052 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:06,054 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,054 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:06,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:06,054 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,054 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:06,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:06,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:06,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:06,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:06,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4607761a[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:06,055 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:06,055 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,057 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:06,058 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,058 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:06,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:06,058 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:06,059 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,059 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,059 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:06,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:06,060 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:06,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:06,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:06,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:06,062 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:06,062 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:06,062 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:06,062 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:06,062 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:06,064 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,064 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:06,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:06,065 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,065 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,066 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,068 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:06,069 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:06,069 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,069 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:06,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:06,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:06,069 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:06,069 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,069 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,069 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,069 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:06,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:06,069 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:06,069 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:06,069 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:06,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45860 deadline: 1681858626114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,125 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/cbc3c4b69d7e43d7beec3143053a2f3b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/cbc3c4b69d7e43d7beec3143053a2f3b 2023-04-18 22:56:06,130 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8181c42622934944fa97cf06c7004a78/A of 8181c42622934944fa97cf06c7004a78 into cbc3c4b69d7e43d7beec3143053a2f3b(size=25.2 K), total size for store is 75.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:06,130 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:06,130 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/A, priority=13, startTime=1681858565235; duration=0sec 2023-04-18 22:56:06,130 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:06,169 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:06,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:06,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:06,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:06,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:06,173 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:06,173 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:06,173 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:06,173 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:06,173 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,173 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,185 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,185 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:06,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:06,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:06,186 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,186 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:06,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:06,187 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:06,187 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:06,187 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:06,187 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:06,187 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:06,187 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:06,187 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:06,187 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:06,187 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:06,187 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:06,187 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:06,187 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,188 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:06,188 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:06,189 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,189 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:06,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:06,190 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,190 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:06,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:06,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:06,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fb29581[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:06,192 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:06,192 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,193 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,193 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:06,194 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:06,194 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:06,194 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,194 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:06,195 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:06,195 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:06,195 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,195 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:06,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:06,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:06,199 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:06,199 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:06,199 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:06,199 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:06,199 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:06,206 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,206 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:06,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,207 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:06,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:06,207 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:06,209 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,211 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:06,211 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:06,211 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,211 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,211 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,211 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,211 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:06,211 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:06,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:06,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:06,211 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:06,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:06,211 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,212 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:06,212 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:06,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:06,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:06,312 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:06,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:06,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:06,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:06,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:06,316 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:06,316 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:06,316 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:06,316 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:06,316 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,316 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,318 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:06,318 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,318 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,318 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:06,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:06,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:06,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:06,319 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:06,319 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:06,319 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:06,320 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:06,320 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:06,320 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:06,320 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:06,320 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:06,320 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:06,320 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:06,320 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:06,320 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,320 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:06,320 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:06,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858626319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,323 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,323 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:06,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:06,323 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,323 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:06,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:06,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:06,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:06,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:06,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ea9cadf[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:06,326 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:06,326 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,327 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,327 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:06,327 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:06,327 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:06,327 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,327 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:06,329 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:06,329 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:06,329 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:06,329 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:06,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:06,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:06,333 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:06,333 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:06,333 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:06,333 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:06,333 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:06,335 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,335 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:06,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:06,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,335 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:06,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:06,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:06,336 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:06,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:06,337 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:06,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,341 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,341 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,341 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:06,341 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:06,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:06,341 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:06,342 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:06,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:06,341 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:06,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:06,342 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:06,342 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,342 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:06,342 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,342 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:06,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:06,442 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:06,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:06,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:06,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:06,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:06,446 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:06,446 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:06,446 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:06,446 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:06,446 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,446 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,447 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:06,447 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:06,448 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,448 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:06,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:06,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:06,449 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:06,449 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:06,449 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:06,449 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:06,449 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:06,449 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:06,449 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:06,449 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:06,449 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:06,449 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:06,449 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:06,449 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,449 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:06,449 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:06,450 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/a618d41a785949e2af596c24ec9dd4cf 2023-04-18 22:56:06,451 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,451 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:06,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:06,451 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,451 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:06,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:06,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:06,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:06,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:06,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@768568f3[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:06,454 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:06,454 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,455 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,455 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:06,455 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:06,455 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:06,456 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,456 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:06,458 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,458 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:06,458 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:06,458 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:06,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:06,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:06,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:06,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,461 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/f338123123cb4267a8b620bb28b1e856 2023-04-18 22:56:06,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:06,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:06,461 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:06,462 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:06,462 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:06,462 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:06,462 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:06,464 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,464 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:06,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:06,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:06,464 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:06,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:06,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,465 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:06,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,466 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/d151bf2efe1d48308a7617c53b2a6029 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/d151bf2efe1d48308a7617c53b2a6029 2023-04-18 22:56:06,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:06,467 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:06,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:06,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,470 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,470 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:06,470 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:06,470 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,470 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,470 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,470 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/d151bf2efe1d48308a7617c53b2a6029, entries=150, sequenceid=174, filesize=25.3 K 2023-04-18 22:56:06,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:06,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:06,470 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,470 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:06,471 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,471 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:06,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:06,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:06,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:06,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:06,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:06,471 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:06,471 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:06,471 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/a618d41a785949e2af596c24ec9dd4cf as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/a618d41a785949e2af596c24ec9dd4cf 2023-04-18 22:56:06,475 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/a618d41a785949e2af596c24ec9dd4cf, entries=150, sequenceid=174, filesize=11.8 K 2023-04-18 22:56:06,475 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/f338123123cb4267a8b620bb28b1e856 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/f338123123cb4267a8b620bb28b1e856 2023-04-18 22:56:06,479 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/f338123123cb4267a8b620bb28b1e856, entries=150, sequenceid=174, filesize=11.8 K 2023-04-18 22:56:06,479 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~265.41 KB/271776, currentSize=87.22 KB/89310 for 8181c42622934944fa97cf06c7004a78 in 551ms, sequenceid=174, compaction requested=true 2023-04-18 22:56:06,479 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:06,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:06,480 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:56:06,480 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-18 22:56:06,480 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:06,480 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:06,480 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-18 22:56:06,480 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-18 22:56:06,480 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. because compaction request was cancelled 2023-04-18 22:56:06,481 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-18 22:56:06,481 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103437 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:56:06,481 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/A is initiating minor compaction (all files) 2023-04-18 22:56:06,481 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/A in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:06,481 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/cbc3c4b69d7e43d7beec3143053a2f3b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/564279ea5ab743579c69e4f7f920680f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/cc878e24d7f24b159221bf03392b1805, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/d151bf2efe1d48308a7617c53b2a6029] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=101.0 K 2023-04-18 22:56:06,481 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-18 22:56:06,481 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-18 22:56:06,481 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. because compaction request was cancelled 2023-04-18 22:56:06,481 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting cbc3c4b69d7e43d7beec3143053a2f3b, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681858564623 2023-04-18 22:56:06,482 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 564279ea5ab743579c69e4f7f920680f, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1681858564852 2023-04-18 22:56:06,482 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting cc878e24d7f24b159221bf03392b1805, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681858565490 2023-04-18 22:56:06,482 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting d151bf2efe1d48308a7617c53b2a6029, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681858565904 2023-04-18 22:56:06,491 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#A#compaction#510 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:06,571 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:06,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:06,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:06,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:06,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:06,575 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:06,575 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:06,575 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:06,575 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:06,575 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,575 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,577 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,577 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:06,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:06,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:06,577 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,577 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:06,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:06,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:06,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:06,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:06,578 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:06,578 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:06,578 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:06,578 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:06,579 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:06,579 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:06,579 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:06,579 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-18 22:56:06,579 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:06,579 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:06,579 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:06,579 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:06,579 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:06,579 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:06,594 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041852a6ecdedd5a45ac9f7ceb3cd63ca6b1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041852a6ecdedd5a45ac9f7ceb3cd63ca6b1 2023-04-18 22:56:06,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:06,625 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:06,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858626644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858626645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858626646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858626647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858626647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858626748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858626751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858626751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858626751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858626754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,913 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/58bd9d6612574846bdbe896a05598eac as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/58bd9d6612574846bdbe896a05598eac 2023-04-18 22:56:06,917 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8181c42622934944fa97cf06c7004a78/A of 8181c42622934944fa97cf06c7004a78 into 58bd9d6612574846bdbe896a05598eac(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:06,917 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:06,918 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/A, priority=12, startTime=1681858566479; duration=0sec 2023-04-18 22:56:06,918 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:06,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858626955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858626955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858626956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858626956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:06,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:06,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858626957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,001 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=193, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/e81e8f9a5b964cb69ddc47793a626d0b 2023-04-18 22:56:07,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:07,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858627259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:07,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858627260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:07,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858627260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:07,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858627261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:07,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858627262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,412 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/f58a5ccc4c1543899ca0acf154b7a9b1 2023-04-18 22:56:07,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:07,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858627762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:07,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858627764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:07,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858627764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:07,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858627765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:07,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858627767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,822 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/261d649e79594bf8b89c0071a05fb974 2023-04-18 22:56:07,827 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/e81e8f9a5b964cb69ddc47793a626d0b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e81e8f9a5b964cb69ddc47793a626d0b 2023-04-18 22:56:07,831 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e81e8f9a5b964cb69ddc47793a626d0b, entries=150, sequenceid=193, filesize=25.3 K 2023-04-18 22:56:07,832 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/f58a5ccc4c1543899ca0acf154b7a9b1 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f58a5ccc4c1543899ca0acf154b7a9b1 2023-04-18 22:56:07,835 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f58a5ccc4c1543899ca0acf154b7a9b1, entries=150, sequenceid=193, filesize=11.8 K 2023-04-18 22:56:07,836 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/261d649e79594bf8b89c0071a05fb974 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/261d649e79594bf8b89c0071a05fb974 2023-04-18 22:56:07,839 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/261d649e79594bf8b89c0071a05fb974, entries=150, sequenceid=193, filesize=11.8 K 2023-04-18 22:56:07,839 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 8181c42622934944fa97cf06c7004a78 in 1260ms, sequenceid=193, compaction requested=true 2023-04-18 22:56:07,839 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:07,839 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:07,840 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:07,840 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:07,840 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,840 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:07,840 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:07,842 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,842 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:07,842 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:07,842 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:07,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:07,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:07,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:07,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:07,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b4f779f[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:07,844 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:07,845 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:07,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:07,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:07,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:07,846 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:07,846 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,846 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:07,846 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:07,846 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:07,847 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,847 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:07,848 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:07,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:07,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:07,848 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:07,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:07,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:07,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:07,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:07,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:07,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:07,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:07,850 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:07,850 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:07,850 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:07,851 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:07,851 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:07,852 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:07,852 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:07,852 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:07,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:07,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:07,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:07,853 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:07,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:07,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:07,854 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:07,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,863 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:07,863 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,863 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:07,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:07,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:07,863 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:07,863 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:07,863 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,863 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:07,863 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:07,863 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:07,863 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:07,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:07,863 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:07,863 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:07,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:07,964 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:07,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:07,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:07,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:07,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:07,968 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:07,968 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:07,968 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:07,968 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:07,968 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:07,968 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:07,970 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:07,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:07,970 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:07,970 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:07,970 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:07,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:07,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:07,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:07,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:07,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:07,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:07,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:07,971 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:07,971 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:07,971 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:07,972 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:07,972 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:07,972 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:07,972 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:07,972 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-18 22:56:07,972 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:07,972 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:07,972 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:07,972 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:07,972 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:07,972 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:07,991 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041834afa4f600194dedb34b5e680bf1831e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041834afa4f600194dedb34b5e680bf1831e 2023-04-18 22:56:08,000 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=211, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/61478ab4cc934eaf8d90955856badb23 2023-04-18 22:56:08,011 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/040adf32319b42f8a6c550d1015aaa3a 2023-04-18 22:56:08,021 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/589d6f0c2cbf471bb3c020da5375017e 2023-04-18 22:56:08,027 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/61478ab4cc934eaf8d90955856badb23 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/61478ab4cc934eaf8d90955856badb23 2023-04-18 22:56:08,031 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/61478ab4cc934eaf8d90955856badb23, entries=150, sequenceid=211, filesize=25.3 K 2023-04-18 22:56:08,031 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/040adf32319b42f8a6c550d1015aaa3a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/040adf32319b42f8a6c550d1015aaa3a 2023-04-18 22:56:08,034 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/040adf32319b42f8a6c550d1015aaa3a, entries=150, sequenceid=211, filesize=11.8 K 2023-04-18 22:56:08,035 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/589d6f0c2cbf471bb3c020da5375017e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/589d6f0c2cbf471bb3c020da5375017e 2023-04-18 22:56:08,039 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/589d6f0c2cbf471bb3c020da5375017e, entries=150, sequenceid=211, filesize=11.8 K 2023-04-18 22:56:08,039 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 8181c42622934944fa97cf06c7004a78 in 67ms, sequenceid=211, compaction requested=true 2023-04-18 22:56:08,039 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:08,039 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:08,040 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:08,040 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:08,040 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,040 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:08,040 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:08,041 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,041 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,042 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,042 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:08,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@500af5ea[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:08,043 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:08,044 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,046 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,046 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:08,046 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:08,046 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:08,056 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:08,056 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:08,056 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:08,056 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,057 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:08,057 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:08,057 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:08,057 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:08,057 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:08,057 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:08,057 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:08,059 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,059 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,059 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:08,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:08,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:08,060 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,061 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,063 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,063 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,063 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,063 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,063 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:08,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:08,063 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:08,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:08,063 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:08,064 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:08,064 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:08,064 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:08,064 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:08,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:08,064 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:08,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:08,164 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:08,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:08,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:08,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:08,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:08,172 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:08,172 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:08,172 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:08,172 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:08,173 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,173 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,175 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:08,175 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:08,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:08,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,176 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,176 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:08,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:08,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:08,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:08,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:08,193 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:08,193 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:08,193 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:08,193 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:08,194 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:08,194 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:08,194 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:08,194 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:08,194 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:08,194 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:08,194 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:08,194 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,194 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:08,194 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:08,196 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,196 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,196 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,196 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:08,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:08,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70d53206[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:08,198 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:08,198 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,199 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,199 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,199 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:08,199 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:08,200 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:08,200 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:08,201 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:08,201 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,201 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:08,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,201 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:08,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:08,204 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:08,204 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:08,204 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:08,204 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:08,204 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:08,207 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,207 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,207 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:08,208 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,210 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,212 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,212 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,212 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:08,213 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:08,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:08,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:08,212 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,213 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,213 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,213 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,213 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,213 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:08,213 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:08,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:08,213 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:08,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:08,313 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:08,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:08,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:08,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:08,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:08,317 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:08,317 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:08,317 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:08,317 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:08,318 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,318 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,319 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:08,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:08,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:08,319 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,320 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,320 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:08,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:08,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:08,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:08,321 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:08,321 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:08,321 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:08,321 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:08,321 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:08,321 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:08,321 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:08,321 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:08,322 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:08,322 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:08,322 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:08,322 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,322 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:08,322 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:08,323 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,323 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,324 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,324 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:08,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d4a0e1f[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:08,326 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:08,326 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,328 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,328 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:08,328 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:08,328 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:08,328 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,328 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:08,330 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,330 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,330 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:08,330 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:08,332 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:08,332 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:08,333 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:08,333 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:08,333 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:08,334 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,334 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,334 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:08,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:08,335 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,336 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:08,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:08,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:08,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:08,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,346 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:08,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:08,346 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:08,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,346 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:08,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,346 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:08,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:08,446 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:08,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:08,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:08,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:08,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:08,450 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:08,450 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:08,450 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:08,450 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:08,451 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,451 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,452 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:08,452 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:08,452 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,452 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:08,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:08,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:08,468 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:08,468 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:08,468 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:08,468 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:08,468 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:08,468 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:08,468 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:08,468 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:08,469 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:08,470 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,470 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:08,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dc977a6[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:08,473 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:08,473 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,480 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,480 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:08,480 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:08,480 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,480 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:08,480 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:08,482 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:08,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:08,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:08,485 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:08,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:08,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:08,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:08,485 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:08,487 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,487 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,487 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:08,488 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,489 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,491 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,491 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,491 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:08,491 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:08,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:08,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:08,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:08,491 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:08,491 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:08,491 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:08,491 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:08,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:08,491 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:08,492 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:08,492 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,492 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,592 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:08,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:08,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:08,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:08,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:08,596 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:08,596 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:08,596 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:08,596 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:08,596 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,597 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,599 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:08,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:08,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:08,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:08,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:08,601 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:08,601 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:08,601 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:08,601 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:08,601 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:08,601 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:08,601 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:08,601 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:08,601 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:08,601 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:08,601 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:08,601 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,601 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:08,601 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:08,603 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,603 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,603 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,603 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:08,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a6b7c50[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:08,605 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:08,605 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,606 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,607 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:08,607 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:08,607 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:08,607 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,607 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:08,608 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,608 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:08,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,608 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:08,608 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:08,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:08,611 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:08,611 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:08,611 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:08,611 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:08,611 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:08,614 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,614 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,614 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:08,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,614 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,616 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,618 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:08,618 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:08,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:08,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:08,618 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,618 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:08,618 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:08,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:08,618 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:08,619 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:08,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:08,619 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:08,619 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:08,619 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,619 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,619 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,719 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:08,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:08,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:08,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:08,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:08,723 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:08,723 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:08,723 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:08,723 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:08,723 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,723 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,724 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,724 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:08,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:08,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:08,725 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,725 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:08,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:08,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:08,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:08,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:08,728 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:08,728 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:08,728 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:08,728 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:08,728 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:08,729 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:08,729 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:08,729 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:08,729 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:08,729 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:08,730 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:08,730 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,730 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:08,730 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:08,731 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,731 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,732 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,732 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:08,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:08,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f73019[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:08,733 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:08,733 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,736 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,736 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:08,736 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:08,736 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:08,736 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,736 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:08,737 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:08,737 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:08,737 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:08,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:08,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:08,740 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:08,740 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:08,740 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:08,740 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:08,740 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:08,742 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,742 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,742 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:08,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:08,743 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,745 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,747 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:08,747 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,747 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:08,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:08,747 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:08,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:08,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:08,747 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:08,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:08,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:08,747 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,748 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,748 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,748 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,748 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,748 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:08,748 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:08,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:08,824 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-18 22:56:08,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:08,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:08,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:08,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:08,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:08,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:08,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:08,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:08,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858628842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858628841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:08,848 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:08,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858628845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,848 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418cd795add7d544e7484c699fb0de0683d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418cd795add7d544e7484c699fb0de0683d 2023-04-18 22:56:08,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:08,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:08,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:08,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858628846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:08,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858628846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:08,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:08,852 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:08,852 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:08,852 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:08,852 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:08,853 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,853 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,855 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:08,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:08,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:08,855 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,855 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,855 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:08,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:08,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:08,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:08,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:08,864 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:08,864 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:08,864 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:08,864 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:08,864 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:08,864 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:08,864 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:08,864 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:08,864 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,864 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:08,864 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:08,869 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,869 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,869 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,870 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:08,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eb0ec8e[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:08,871 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:08,871 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,872 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,872 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:08,872 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,873 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:08,873 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:08,873 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:08,874 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:08,874 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:08,875 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:08,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:08,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:08,877 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:08,877 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:08,877 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:08,877 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:08,877 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:08,880 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,880 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:08,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:08,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:08,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,884 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:08,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:08,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:08,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:08,885 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:08,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:08,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,887 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:08,890 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,890 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:08,891 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:08,891 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:08,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:08,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:08,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:08,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:08,891 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:08,891 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:08,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:08,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:08,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:08,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858628947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:08,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858628947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:08,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858628949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:08,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858628952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:08,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858628952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,992 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:08,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:08,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:08,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:08,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:08,995 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:08,995 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:08,995 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:08,995 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:08,996 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,996 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,997 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:08,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:08,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:08,998 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,998 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:08,998 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:08,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:08,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:08,999 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:08,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:08,999 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:08,999 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:08,999 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:08,999 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:08,999 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:08,999 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:09,000 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:09,000 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:09,000 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:09,000 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:09,000 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:09,000 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,000 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:09,000 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:09,002 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,002 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,002 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,002 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:09,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:09,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68d2c0b9[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:09,004 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:09,005 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,006 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,007 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:09,007 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:09,007 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:09,007 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,007 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:09,009 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:09,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,009 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:09,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:09,013 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:09,013 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:09,013 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:09,013 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:09,013 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:09,015 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,015 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,015 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:09,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:09,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:09,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,015 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,017 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:09,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:09,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:09,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:09,020 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:09,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,020 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:09,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:09,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:09,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:09,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:09,021 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:09,021 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:09,121 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:09,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:09,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:09,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:09,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:09,124 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:09,124 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:09,124 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:09,124 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:09,125 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,125 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,127 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:09,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:09,127 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,128 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,128 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:09,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:09,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:09,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:09,129 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:09,129 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:09,129 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:09,129 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:09,129 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:09,129 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:09,129 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:09,129 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:09,129 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:09,129 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:09,129 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:09,129 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,129 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:09,129 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:09,131 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,131 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,131 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,131 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:09,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:09,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@695aa80f[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:09,132 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:09,133 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,134 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:09,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:09,134 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:09,134 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,134 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:09,135 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:09,135 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:09,135 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:09,135 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:09,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:09,138 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:09,138 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:09,138 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:09,138 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:09,138 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:09,140 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,141 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:09,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,141 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:09,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:09,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,141 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,143 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,145 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,145 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,145 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:09,145 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:09,145 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:09,145 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:09,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:09,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:09,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:09,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:09,145 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,145 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,145 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,145 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,145 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:09,145 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:09,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:09,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:09,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:09,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858629152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:09,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858629153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:09,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858629153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:09,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858629157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:09,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858629157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,246 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:09,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:09,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:09,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:09,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:09,250 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:09,250 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:09,250 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:09,250 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:09,250 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,250 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,252 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:09,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:09,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:09,252 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,252 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,252 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:09,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:09,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:09,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:09,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:09,254 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:09,254 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:09,254 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:09,256 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:09,256 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:09,256 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:09,256 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:09,256 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:09,256 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:09,256 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:09,256 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:09,256 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,256 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:09,256 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:09,258 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,259 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=229, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/f5ac854c2d4740bea8f8d93af41a6787 2023-04-18 22:56:09,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:09,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53173ac5[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:09,261 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:09,262 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,267 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,267 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:09,267 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:09,267 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:09,267 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,267 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:09,269 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,269 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:09,269 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:09,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,269 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:09,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:09,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:09,273 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:09,273 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:09,273 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:09,273 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:09,273 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:09,275 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,275 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,276 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:09,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:09,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:09,276 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,278 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,281 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,281 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,281 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,281 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:09,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:09,281 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:09,281 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,281 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:09,281 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:09,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:09,281 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:09,281 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,281 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:09,282 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:09,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:09,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:09,382 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:09,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:09,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:09,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:09,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:09,385 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:09,385 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:09,385 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:09,385 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:09,386 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,386 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,387 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:09,387 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:09,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:09,387 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,387 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:09,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:09,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:09,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:09,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:09,388 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:09,388 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:09,388 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:09,392 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:09,392 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:09,392 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:09,392 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:09,392 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:09,392 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:09,392 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:09,392 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:09,392 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,392 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:09,392 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:09,394 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,394 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,394 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,394 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:09,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:09,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aba68b5[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:09,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:09,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,397 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,397 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,397 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:09,397 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:09,397 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:09,397 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:09,399 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,399 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,399 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:09,399 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:09,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:09,403 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:09,403 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:09,403 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:09,403 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:09,403 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:09,405 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,405 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:09,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:09,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:09,405 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,406 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,407 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,409 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,409 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,409 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:09,409 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:09,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:09,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:09,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:09,409 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:09,409 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,409 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,409 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,409 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,409 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:09,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:09,409 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:09,410 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:09,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:09,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858629457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:09,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858629459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:09,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858629459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:09,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858629459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:09,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858629461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,510 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:09,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:09,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:09,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:09,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:09,513 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:09,513 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:09,513 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:09,513 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:09,514 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,514 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,515 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,515 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:09,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:09,516 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,516 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:09,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:09,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:09,517 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:09,517 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:09,517 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:09,517 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:09,517 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:09,517 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:09,518 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:09,518 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:09,518 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:09,518 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:09,518 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:09,518 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,518 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:09,518 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:09,520 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,520 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,520 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,520 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:09,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5413a253[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:09,522 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:09,522 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,523 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,523 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:09,523 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:09,523 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,524 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:09,523 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:09,525 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:09,525 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:09,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,525 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:09,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:09,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:09,528 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:09,528 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:09,528 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:09,528 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:09,528 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:09,529 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,529 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,529 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:09,530 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,532 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,541 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,541 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,541 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:09,541 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:09,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:09,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:09,541 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,541 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:09,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:09,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:09,541 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:09,541 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:09,541 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,541 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:09,541 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,542 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:09,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:09,642 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:09,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:09,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:09,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:09,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:09,645 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:09,646 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:09,646 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:09,646 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:09,646 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,646 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,647 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:09,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:09,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:09,648 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,648 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,648 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:09,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:09,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:09,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:09,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:09,649 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:09,649 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:09,649 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:09,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:09,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:09,652 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:09,652 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:09,652 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:09,652 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:09,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:09,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:09,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:09,652 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:09,654 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,654 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,654 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,654 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:09,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:09,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6835720e[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:09,656 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:09,656 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,657 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,658 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:09,658 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:09,658 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:09,658 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,658 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:09,661 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:09,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:09,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,661 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:09,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:09,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:09,664 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:09,664 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:09,664 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:09,664 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:09,664 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:09,669 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:09,669 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,669 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:09,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:09,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,670 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:09,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,671 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:09,672 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/f0c7e83aa4fd4469b44fcb59de72176c 2023-04-18 22:56:09,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:09,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,675 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,675 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,675 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,675 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,675 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,675 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:09,675 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,675 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:09,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:09,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:09,675 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:09,675 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:09,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:09,676 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:09,676 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:09,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:09,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:09,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:09,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:09,686 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/2cc42a5d68ca4e15a95b05eaf09939c1 2023-04-18 22:56:09,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/f5ac854c2d4740bea8f8d93af41a6787 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/f5ac854c2d4740bea8f8d93af41a6787 2023-04-18 22:56:09,694 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/f5ac854c2d4740bea8f8d93af41a6787, entries=200, sequenceid=229, filesize=32.1 K 2023-04-18 22:56:09,695 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/f0c7e83aa4fd4469b44fcb59de72176c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f0c7e83aa4fd4469b44fcb59de72176c 2023-04-18 22:56:09,699 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f0c7e83aa4fd4469b44fcb59de72176c, entries=150, sequenceid=229, filesize=11.8 K 2023-04-18 22:56:09,700 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/2cc42a5d68ca4e15a95b05eaf09939c1 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/2cc42a5d68ca4e15a95b05eaf09939c1 2023-04-18 22:56:09,705 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/2cc42a5d68ca4e15a95b05eaf09939c1, entries=150, sequenceid=229, filesize=11.8 K 2023-04-18 22:56:09,705 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 8181c42622934944fa97cf06c7004a78 in 881ms, sequenceid=229, compaction requested=true 2023-04-18 22:56:09,705 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:09,706 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:09,706 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:56:09,706 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:09,706 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 22:56:09,706 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:09,707 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110814 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:56:09,707 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/A is initiating minor compaction (all files) 2023-04-18 22:56:09,707 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/A in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:09,707 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/58bd9d6612574846bdbe896a05598eac, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e81e8f9a5b964cb69ddc47793a626d0b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/61478ab4cc934eaf8d90955856badb23, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/f5ac854c2d4740bea8f8d93af41a6787] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=108.2 K 2023-04-18 22:56:09,708 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60386 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 22:56:09,708 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/B is initiating minor compaction (all files) 2023-04-18 22:56:09,708 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 58bd9d6612574846bdbe896a05598eac, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681858565904 2023-04-18 22:56:09,708 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/B in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:09,708 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/5364433954ce4561814e30819c239fb8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/a618d41a785949e2af596c24ec9dd4cf, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f58a5ccc4c1543899ca0acf154b7a9b1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/040adf32319b42f8a6c550d1015aaa3a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f0c7e83aa4fd4469b44fcb59de72176c] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=59.0 K 2023-04-18 22:56:09,709 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting e81e8f9a5b964cb69ddc47793a626d0b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681858565983 2023-04-18 22:56:09,709 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 5364433954ce4561814e30819c239fb8, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681858565490 2023-04-18 22:56:09,709 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 61478ab4cc934eaf8d90955856badb23, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681858566644 2023-04-18 22:56:09,709 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting a618d41a785949e2af596c24ec9dd4cf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681858565904 2023-04-18 22:56:09,709 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting f5ac854c2d4740bea8f8d93af41a6787, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681858568791 2023-04-18 22:56:09,710 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting f58a5ccc4c1543899ca0acf154b7a9b1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681858565983 2023-04-18 22:56:09,710 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 040adf32319b42f8a6c550d1015aaa3a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681858566644 2023-04-18 22:56:09,710 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting f0c7e83aa4fd4469b44fcb59de72176c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681858568792 2023-04-18 22:56:09,725 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#B#compaction#521 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:09,725 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#A#compaction#520 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:09,759 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/98045c6babbe401b86fc3ca2a22a9ba4 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/98045c6babbe401b86fc3ca2a22a9ba4 2023-04-18 22:56:09,764 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8181c42622934944fa97cf06c7004a78/B of 8181c42622934944fa97cf06c7004a78 into 98045c6babbe401b86fc3ca2a22a9ba4(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:09,764 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:09,764 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/B, priority=11, startTime=1681858569706; duration=0sec 2023-04-18 22:56:09,764 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:09,764 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 22:56:09,765 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60386 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 22:56:09,766 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/C is initiating minor compaction (all files) 2023-04-18 22:56:09,766 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/C in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:09,766 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/3bc6d813bc334a5f8e3215b43aafebd3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/f338123123cb4267a8b620bb28b1e856, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/261d649e79594bf8b89c0071a05fb974, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/589d6f0c2cbf471bb3c020da5375017e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/2cc42a5d68ca4e15a95b05eaf09939c1] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=59.0 K 2023-04-18 22:56:09,766 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 3bc6d813bc334a5f8e3215b43aafebd3, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681858565490 2023-04-18 22:56:09,767 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting f338123123cb4267a8b620bb28b1e856, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681858565904 2023-04-18 22:56:09,767 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 261d649e79594bf8b89c0071a05fb974, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681858565983 2023-04-18 22:56:09,767 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 589d6f0c2cbf471bb3c020da5375017e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1681858566644 2023-04-18 22:56:09,768 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 2cc42a5d68ca4e15a95b05eaf09939c1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681858568792 2023-04-18 22:56:09,776 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:09,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:09,778 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#C#compaction#522 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:09,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:09,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:09,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:09,789 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:09,789 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:09,789 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:09,789 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:09,789 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,789 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,791 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:09,791 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:09,791 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,791 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:09,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:09,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:09,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:09,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:09,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:09,800 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:09,800 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:09,800 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:09,816 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:09,816 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:09,816 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:09,816 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:09,817 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-18 22:56:09,817 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:09,817 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:09,817 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:09,817 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:09,817 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:09,817 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:09,821 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/e28f02b6eb3b4d6bae1d23b65c5562f7 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/e28f02b6eb3b4d6bae1d23b65c5562f7 2023-04-18 22:56:09,830 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8181c42622934944fa97cf06c7004a78/C of 8181c42622934944fa97cf06c7004a78 into e28f02b6eb3b4d6bae1d23b65c5562f7(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:09,830 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:09,830 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/C, priority=11, startTime=1681858569706; duration=0sec 2023-04-18 22:56:09,830 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:09,964 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:09,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:09,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:09,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858629969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:09,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:09,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858629971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858629971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:09,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858629973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:09,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:09,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858629973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:10,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:10,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858630074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:10,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:10,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858630075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:10,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:10,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858630076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:10,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:10,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858630078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:10,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:10,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858630078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:10,165 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/828ac514d2364cb0ae4a58846167e982 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/828ac514d2364cb0ae4a58846167e982 2023-04-18 22:56:10,170 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8181c42622934944fa97cf06c7004a78/A of 8181c42622934944fa97cf06c7004a78 into 828ac514d2364cb0ae4a58846167e982(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:10,170 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:10,170 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/A, priority=12, startTime=1681858569705; duration=0sec 2023-04-18 22:56:10,170 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:10,257 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304187fceea7256cd4a46a8810c6cfc8f004c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187fceea7256cd4a46a8810c6cfc8f004c 2023-04-18 22:56:10,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:10,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858630278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:10,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:10,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858630278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:10,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:10,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858630280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:10,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:10,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858630281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:10,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:10,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858630281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:10,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:10,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858630580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:10,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:10,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858630581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:10,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:10,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858630584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:10,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:10,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858630584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:10,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:10,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858630585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:10,666 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=249, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/d7c662601feb47569d1d124531c4bca0 2023-04-18 22:56:10,675 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/345b5b72331645ba9e642879161de145 2023-04-18 22:56:11,084 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/5128c555dc1d40c2b2137acd8184d9ea 2023-04-18 22:56:11,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:11,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858631087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:11,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858631087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:11,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858631087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:11,089 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/d7c662601feb47569d1d124531c4bca0 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/d7c662601feb47569d1d124531c4bca0 2023-04-18 22:56:11,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858631087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:11,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858631090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,093 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/d7c662601feb47569d1d124531c4bca0, entries=150, sequenceid=249, filesize=25.3 K 2023-04-18 22:56:11,094 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/345b5b72331645ba9e642879161de145 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/345b5b72331645ba9e642879161de145 2023-04-18 22:56:11,097 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/345b5b72331645ba9e642879161de145, entries=150, sequenceid=249, filesize=11.8 K 2023-04-18 22:56:11,098 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/5128c555dc1d40c2b2137acd8184d9ea as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/5128c555dc1d40c2b2137acd8184d9ea 2023-04-18 22:56:11,105 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/5128c555dc1d40c2b2137acd8184d9ea, entries=150, sequenceid=249, filesize=11.8 K 2023-04-18 22:56:11,106 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 8181c42622934944fa97cf06c7004a78 in 1290ms, sequenceid=249, compaction requested=false 2023-04-18 22:56:11,106 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:11,106 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:11,106 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:11,106 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:11,106 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,106 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:11,106 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:11,108 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:11,108 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:11,109 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,109 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:11,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:11,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29ebcdf[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:11,110 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:11,110 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,111 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,112 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:11,112 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:11,112 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:11,112 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,112 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:11,113 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,114 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:11,114 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:11,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:11,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:11,114 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:11,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:11,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:11,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:11,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:11,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:11,117 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:11,117 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:11,117 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:11,117 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:11,117 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:11,120 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:11,120 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,120 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:11,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:11,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:11,120 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,122 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:11,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:11,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:11,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:11,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:11,124 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:11,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:11,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:11,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:11,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:11,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,124 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:11,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:11,125 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:11,125 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:11,225 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:11,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:11,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:11,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:11,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:11,228 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:11,228 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:11,228 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:11,229 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:11,229 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,229 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:11,230 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:11,230 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:11,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:11,231 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,231 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:11,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:11,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:11,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:11,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:11,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:11,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:11,232 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:11,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:11,233 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:11,236 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:11,236 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:11,236 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:11,236 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:11,236 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-18 22:56:11,236 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:11,236 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:11,237 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:11,237 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:11,237 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:11,237 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:11,257 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041859cae9e13ca647cba88b0060164ceeb6 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041859cae9e13ca647cba88b0060164ceeb6 2023-04-18 22:56:11,666 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=265, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/c5ade2f2104a41b38430ebe872c3f72f 2023-04-18 22:56:11,676 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/c1203a835680480ebc4d432c3ba5900a 2023-04-18 22:56:11,686 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/9f0bf10a3ea44b8c8500d1826f2ce748 2023-04-18 22:56:11,690 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/c5ade2f2104a41b38430ebe872c3f72f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/c5ade2f2104a41b38430ebe872c3f72f 2023-04-18 22:56:11,694 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/c5ade2f2104a41b38430ebe872c3f72f, entries=150, sequenceid=265, filesize=25.4 K 2023-04-18 22:56:11,695 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/c1203a835680480ebc4d432c3ba5900a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/c1203a835680480ebc4d432c3ba5900a 2023-04-18 22:56:11,699 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/c1203a835680480ebc4d432c3ba5900a, entries=150, sequenceid=265, filesize=11.9 K 2023-04-18 22:56:11,699 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/9f0bf10a3ea44b8c8500d1826f2ce748 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/9f0bf10a3ea44b8c8500d1826f2ce748 2023-04-18 22:56:11,706 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/9f0bf10a3ea44b8c8500d1826f2ce748, entries=150, sequenceid=265, filesize=11.9 K 2023-04-18 22:56:11,707 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 8181c42622934944fa97cf06c7004a78 in 471ms, sequenceid=265, compaction requested=true 2023-04-18 22:56:11,707 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:11,707 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:11,707 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:11,707 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:11,707 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,707 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:11,707 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:11,709 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,709 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:11,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:11,709 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,709 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:11,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:11,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:11,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:11,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:11,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b1893f9[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:11,711 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:11,711 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,713 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,713 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:11,713 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:11,713 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:11,714 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,714 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:11,717 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,717 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:11,717 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:11,717 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:11,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:11,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:11,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:11,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:11,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:11,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:11,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:11,720 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:11,720 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:11,720 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:11,720 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:11,720 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:11,722 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,722 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:11,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,722 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:11,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:11,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:11,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:11,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:11,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,723 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:11,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,724 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:11,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,727 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:11,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:11,727 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:11,727 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:11,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:11,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:11,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:11,727 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:11,727 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:11,728 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:11,727 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:11,728 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:11,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:11,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:11,728 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,728 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,728 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,728 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,828 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:11,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:11,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:11,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:11,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:11,831 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:11,831 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:11,831 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:11,831 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:11,832 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,832 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:11,834 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,834 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:11,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:11,835 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,835 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:11,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:11,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:11,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:11,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:11,836 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:11,836 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:11,836 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:11,836 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:11,836 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:11,836 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:11,836 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:11,836 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:11,836 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:11,837 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:11,837 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:11,837 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,837 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:11,837 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:11,838 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,838 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:11,839 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,839 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:11,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:11,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3250b7f3[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:11,840 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:11,840 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,842 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,842 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:11,842 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:11,842 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:11,842 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,842 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:11,843 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:11,843 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,843 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:11,843 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:11,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:11,846 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:11,846 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:11,846 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:11,846 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:11,846 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:11,849 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,849 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:11,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,849 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:11,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:11,849 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:11,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,851 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,856 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:11,856 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:11,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:11,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:11,856 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,856 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:11,856 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:11,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:11,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:11,856 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,856 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,856 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,856 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,856 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:11,856 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:11,856 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:11,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:11,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:11,957 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:11,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:11,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:11,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:11,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:11,960 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:11,960 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:11,961 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:11,961 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:11,961 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,961 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:11,962 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,962 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:11,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:11,962 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,963 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:11,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:11,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:11,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:11,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:11,964 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:11,964 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:11,964 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:11,964 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:11,964 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:11,964 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:11,964 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:11,964 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:11,964 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:11,965 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:11,965 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:11,965 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,965 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:11,965 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:11,966 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,966 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:11,966 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:11,966 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:11,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:11,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71b606b5[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:11,968 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:11,968 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,969 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,970 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:11,970 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:11,970 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,970 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:11,970 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:11,972 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,972 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:11,972 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:11,972 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:11,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:11,975 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:11,975 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:11,975 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:11,975 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:11,975 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:11,977 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,977 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:11,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:11,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,977 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:11,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:11,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:11,977 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:11,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,979 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,981 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,981 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:11,981 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:11,981 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:11,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:11,981 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:11,981 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:11,981 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,981 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:11,981 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:11,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:11,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:11,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:11,981 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:11,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:11,982 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:11,982 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:11,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:11,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:12,082 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:12,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:12,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:12,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:12,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:12,086 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:12,086 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:12,086 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:12,086 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:12,086 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,086 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,088 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,089 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:12,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:12,089 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,089 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:12,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:12,090 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:12,090 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:12,090 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:12,090 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:12,090 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:12,090 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:12,091 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:12,091 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:12,091 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:12,091 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:12,091 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:12,091 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:12,091 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,091 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:12,091 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:12,095 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,095 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:12,095 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,095 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:12,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:12,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b92418[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:12,097 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:12,097 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,098 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,098 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:12,098 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:12,099 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:12,099 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,099 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:12,116 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,116 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:12,116 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:12,116 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:12,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:12,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:12,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:12,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:12,119 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:12,119 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:12,119 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:12,119 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:12,119 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:12,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:12,121 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,122 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:12,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:12,123 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,127 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,127 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:12,128 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:12,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:12,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:12,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:12,128 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:12,128 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:12,128 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:12,127 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:12,128 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,128 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,128 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,128 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,128 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:12,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:12,141 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:56:12,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:12,141 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:12,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:12,141 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:12,141 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:12,141 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:12,153 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418f68db1e3c2d24c49ba3ba788afbdb547 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f68db1e3c2d24c49ba3ba788afbdb547 2023-04-18 22:56:12,158 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=278, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/fd2aa31b472f4f10b07694edbf132a3f 2023-04-18 22:56:12,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858572159, max timestamp=1681858572163Num uniques 859; 2023-04-18 22:56:12,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:56:12,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858572159, max timestamp=1681858572163Num uniques 859; 2023-04-18 22:56:12,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:56:12,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858572159, max timestamp=1681858572163Num uniques 859; 2023-04-18 22:56:12,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:56:12,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/42bc8ad60e84413e9138abe4e1d737dc 2023-04-18 22:56:12,179 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/d7fc7719ec1c4d9c9ad140c97dad48ae 2023-04-18 22:56:12,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/fd2aa31b472f4f10b07694edbf132a3f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/fd2aa31b472f4f10b07694edbf132a3f 2023-04-18 22:56:12,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/fd2aa31b472f4f10b07694edbf132a3f, entries=250, sequenceid=278, filesize=39.2 K 2023-04-18 22:56:12,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/42bc8ad60e84413e9138abe4e1d737dc as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/42bc8ad60e84413e9138abe4e1d737dc 2023-04-18 22:56:12,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858632181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,190 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/42bc8ad60e84413e9138abe4e1d737dc, entries=150, sequenceid=278, filesize=11.9 K 2023-04-18 22:56:12,190 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/d7fc7719ec1c4d9c9ad140c97dad48ae as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/d7fc7719ec1c4d9c9ad140c97dad48ae 2023-04-18 22:56:12,193 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/d7fc7719ec1c4d9c9ad140c97dad48ae, entries=150, sequenceid=278, filesize=11.9 K 2023-04-18 22:56:12,194 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 8181c42622934944fa97cf06c7004a78 in 54ms, sequenceid=278, compaction requested=true 2023-04-18 22:56:12,194 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:12,194 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:12,194 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:56:12,194 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:56:12,194 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:12,194 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:12,195 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 118344 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:56:12,196 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/A is initiating minor compaction (all files) 2023-04-18 22:56:12,196 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/A in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:12,196 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/828ac514d2364cb0ae4a58846167e982, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/d7c662601feb47569d1d124531c4bca0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/c5ade2f2104a41b38430ebe872c3f72f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/fd2aa31b472f4f10b07694edbf132a3f] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=115.6 K 2023-04-18 22:56:12,196 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48820 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:56:12,196 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/B is initiating minor compaction (all files) 2023-04-18 22:56:12,196 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/B in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:12,196 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 828ac514d2364cb0ae4a58846167e982, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681858568792 2023-04-18 22:56:12,196 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/98045c6babbe401b86fc3ca2a22a9ba4, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/345b5b72331645ba9e642879161de145, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/c1203a835680480ebc4d432c3ba5900a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/42bc8ad60e84413e9138abe4e1d737dc] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=47.7 K 2023-04-18 22:56:12,197 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting d7c662601feb47569d1d124531c4bca0, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681858568838 2023-04-18 22:56:12,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:12,197 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=154.31 KB heapSize=440.77 KB 2023-04-18 22:56:12,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7682, totalHeapSize=17712, min timestamp=1681858572186, max timestamp=1681858572196Num uniques 1068; 2023-04-18 22:56:12,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48902, new segment data size=7682, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-18 22:56:12,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7682, totalHeapSize=17712, min timestamp=1681858572186, max timestamp=1681858572196Num uniques 1068; 2023-04-18 22:56:12,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48902, new segment data size=7682, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-18 22:56:12,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858572186, max timestamp=1681858572195Num uniques 1068; 2023-04-18 22:56:12,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-18 22:56:12,199 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 98045c6babbe401b86fc3ca2a22a9ba4, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681858568792 2023-04-18 22:56:12,200 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting c5ade2f2104a41b38430ebe872c3f72f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681858569969 2023-04-18 22:56:12,200 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 345b5b72331645ba9e642879161de145, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681858568838 2023-04-18 22:56:12,200 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting fd2aa31b472f4f10b07694edbf132a3f, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1681858572121 2023-04-18 22:56:12,200 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting c1203a835680480ebc4d432c3ba5900a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681858569969 2023-04-18 22:56:12,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:12,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:12,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:12,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:12,201 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 42bc8ad60e84413e9138abe4e1d737dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1681858572121 2023-04-18 22:56:12,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:12,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:12,228 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:12,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:12,229 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#B#compaction#532 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:12,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858572206, max timestamp=1681858572228Num uniques 859; 2023-04-18 22:56:12,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:56:12,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858572206, max timestamp=1681858572228Num uniques 859; 2023-04-18 22:56:12,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:56:12,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858572206, max timestamp=1681858572228Num uniques 859; 2023-04-18 22:56:12,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:56:12,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:12,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:12,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:12,240 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:12,240 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:12,240 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:12,240 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:12,241 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,241 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,242 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,242 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:12,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:12,242 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,242 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:12,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:12,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:12,245 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:12,245 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:12,245 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:12,246 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:12,246 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:12,246 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:12,246 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:12,246 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:12,246 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:12,246 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:12,246 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:12,246 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,248 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:12,248 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:12,250 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418f9ea491021014147bb703ce61f4aa861 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f9ea491021014147bb703ce61f4aa861 2023-04-18 22:56:12,250 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,250 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,250 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:12,250 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:12,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:12,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:12,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3901e86d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:12,254 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:12,254 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,254 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#A#compaction#534 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:12,255 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,255 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:12,255 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:12,255 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:12,256 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,256 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:12,256 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=319, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/9ecd515dae9e48ca8abdf72718dc331f 2023-04-18 22:56:12,257 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:12,257 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:12,257 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:12,257 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:12,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:12,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858632253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858632254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:12,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:12,261 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:12,261 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:12,261 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:12,261 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:12,261 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:12,263 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,263 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,263 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:12,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:12,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:12,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,264 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:12,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:12,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:12,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:12,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,265 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:12,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858632260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858632261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,270 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:12,270 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:12,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:12,270 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:12,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:12,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:12,271 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:12,270 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,271 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,271 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,271 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,271 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,271 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,271 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:12,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:12,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:12,271 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:12,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:12,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858632290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858632361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858632361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858632367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858632367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,371 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:12,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:12,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:12,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:12,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:12,375 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:12,375 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:12,375 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:12,375 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:12,375 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,375 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,377 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:12,377 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:12,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:12,378 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,378 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:12,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,378 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:12,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:12,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:12,379 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:12,379 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:12,379 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:12,379 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:12,381 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:12,381 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:12,382 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,382 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:12,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:12,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ed93f91[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:12,383 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:12,383 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,384 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,384 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:12,384 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:12,384 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:12,384 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,384 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:12,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:12,386 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:12,385 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,386 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:12,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:12,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:12,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:12,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:12,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:12,389 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:12,390 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:12,390 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:12,390 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:12,390 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:12,392 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,392 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,392 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:12,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:12,393 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:12,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,394 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,396 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,396 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,396 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:12,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:12,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:12,396 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:12,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:12,396 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,397 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,396 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:12,397 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,397 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:12,397 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:12,397 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,397 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:12,497 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:12,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:12,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858632495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:12,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:12,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:12,501 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:12,501 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:12,501 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:12,501 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:12,501 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,502 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,503 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,503 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:12,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:12,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:12,503 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,503 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:12,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:12,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:12,505 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:12,505 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:12,505 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:12,505 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:12,505 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:12,505 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:12,505 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:12,505 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:12,505 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:12,505 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:12,505 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,505 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:12,505 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:12,507 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,507 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:12,507 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,507 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:12,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:12,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f1b0351[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:12,509 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:12,509 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,511 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,512 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:12,512 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:12,512 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:12,512 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,512 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:12,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:12,513 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:12,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:12,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:12,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:12,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:12,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:12,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:12,515 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:12,516 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:12,516 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:12,516 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:12,516 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:12,517 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,517 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,517 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:12,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:12,518 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:12,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:12,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:12,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,521 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:12,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:12,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:12,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:12,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:12,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:12,521 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,522 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,522 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,522 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,522 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:12,522 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:12,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:12,522 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:12,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:12,522 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858632565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858632565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858632572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858632572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,622 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:12,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:12,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:12,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:12,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:12,625 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:12,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:12,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:12,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:12,626 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,626 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,628 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,628 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:12,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:12,628 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,628 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:12,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:12,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:12,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:12,629 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:12,630 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:12,630 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:12,630 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:12,630 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:12,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,632 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:12,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,632 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:12,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:12,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:12,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:12,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:12,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cd674e[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:12,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:12,635 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,636 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,636 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:12,636 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:12,636 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:12,636 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,636 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:12,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:12,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:12,638 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:12,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:12,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:12,641 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:12,641 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:12,641 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:12,641 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:12,641 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:12,643 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:12,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:12,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:12,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:12,644 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:12,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:12,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:12,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,646 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:12,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:12,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,648 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:12,648 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:12,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:12,648 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:12,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:12,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:12,648 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:12,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:12,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:12,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:12,649 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:12,649 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:12,648 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,649 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,649 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,649 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,649 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:12,649 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,654 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/f581499d082d4841bf4fefc4da276d0e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f581499d082d4841bf4fefc4da276d0e 2023-04-18 22:56:12,658 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8181c42622934944fa97cf06c7004a78/B of 8181c42622934944fa97cf06c7004a78 into f581499d082d4841bf4fefc4da276d0e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:12,658 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:12,658 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/B, priority=12, startTime=1681858572194; duration=0sec 2023-04-18 22:56:12,658 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:12,658 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:56:12,659 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48820 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:56:12,659 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/C is initiating minor compaction (all files) 2023-04-18 22:56:12,659 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/C in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:12,660 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/e28f02b6eb3b4d6bae1d23b65c5562f7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/5128c555dc1d40c2b2137acd8184d9ea, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/9f0bf10a3ea44b8c8500d1826f2ce748, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/d7fc7719ec1c4d9c9ad140c97dad48ae] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=47.7 K 2023-04-18 22:56:12,660 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting e28f02b6eb3b4d6bae1d23b65c5562f7, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681858568792 2023-04-18 22:56:12,660 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 5128c555dc1d40c2b2137acd8184d9ea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681858568838 2023-04-18 22:56:12,660 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 9f0bf10a3ea44b8c8500d1826f2ce748, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681858569969 2023-04-18 22:56:12,661 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting d7fc7719ec1c4d9c9ad140c97dad48ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1681858572121 2023-04-18 22:56:12,668 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#C#compaction#536 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:12,678 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/e5db8fe6d5f74b73878b54af650f6257 2023-04-18 22:56:12,685 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/49d5072797b84e61b9178384f0e33263 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/49d5072797b84e61b9178384f0e33263 2023-04-18 22:56:12,698 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8181c42622934944fa97cf06c7004a78/C of 8181c42622934944fa97cf06c7004a78 into 49d5072797b84e61b9178384f0e33263(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:12,698 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:12,698 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/C, priority=12, startTime=1681858572194; duration=0sec 2023-04-18 22:56:12,698 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:12,715 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/43e7a9ffc77641dab32944f4f2a2b324 2023-04-18 22:56:12,716 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/4ee4ba3abe8b402dbd123d83388d7c60 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4ee4ba3abe8b402dbd123d83388d7c60 2023-04-18 22:56:12,719 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/9ecd515dae9e48ca8abdf72718dc331f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/9ecd515dae9e48ca8abdf72718dc331f 2023-04-18 22:56:12,720 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8181c42622934944fa97cf06c7004a78/A of 8181c42622934944fa97cf06c7004a78 into 4ee4ba3abe8b402dbd123d83388d7c60(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:12,720 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:12,720 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/A, priority=12, startTime=1681858572194; duration=0sec 2023-04-18 22:56:12,721 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:12,722 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/9ecd515dae9e48ca8abdf72718dc331f, entries=250, sequenceid=319, filesize=39.2 K 2023-04-18 22:56:12,723 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/e5db8fe6d5f74b73878b54af650f6257 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/e5db8fe6d5f74b73878b54af650f6257 2023-04-18 22:56:12,726 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/e5db8fe6d5f74b73878b54af650f6257, entries=150, sequenceid=319, filesize=11.9 K 2023-04-18 22:56:12,727 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/43e7a9ffc77641dab32944f4f2a2b324 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/43e7a9ffc77641dab32944f4f2a2b324 2023-04-18 22:56:12,730 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/43e7a9ffc77641dab32944f4f2a2b324, entries=150, sequenceid=319, filesize=11.9 K 2023-04-18 22:56:12,730 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~125.11 KB/128112, currentSize=147.60 KB/151140 for 8181c42622934944fa97cf06c7004a78 in 533ms, sequenceid=319, compaction requested=false 2023-04-18 22:56:12,730 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:12,749 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:12,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:12,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:12,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:12,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:12,753 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:12,753 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:12,753 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:12,753 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:12,753 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,753 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,754 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,754 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:12,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:12,755 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,755 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:12,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:12,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:12,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:12,756 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:12,756 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:12,756 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:12,756 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:12,756 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:12,757 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:12,757 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:12,757 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-18 22:56:12,757 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:12,757 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:12,757 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:12,757 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:12,757 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:12,757 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:12,773 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041820f46ad84fbe445f9ea833bbba396544 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041820f46ad84fbe445f9ea833bbba396544 2023-04-18 22:56:12,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:12,802 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:12,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858632814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858632869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858632869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858632876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858632877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:12,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:12,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858632917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:13,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858633121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,182 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=359, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/ad9466a783bd4113b5f94da704bb25d5 2023-04-18 22:56:13,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:13,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858633373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:13,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858633374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:13,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858633381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:13,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858633382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:13,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45852 deadline: 1681858633425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,592 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/88a612595ce84c21b710bef84803aea2 2023-04-18 22:56:13,602 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/a73bf9fe2ea34f4d86c40f120399f70b 2023-04-18 22:56:13,605 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/ad9466a783bd4113b5f94da704bb25d5 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/ad9466a783bd4113b5f94da704bb25d5 2023-04-18 22:56:13,609 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/ad9466a783bd4113b5f94da704bb25d5, entries=150, sequenceid=359, filesize=25.4 K 2023-04-18 22:56:13,610 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/88a612595ce84c21b710bef84803aea2 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/88a612595ce84c21b710bef84803aea2 2023-04-18 22:56:13,613 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/88a612595ce84c21b710bef84803aea2, entries=150, sequenceid=359, filesize=11.9 K 2023-04-18 22:56:13,614 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/a73bf9fe2ea34f4d86c40f120399f70b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/a73bf9fe2ea34f4d86c40f120399f70b 2023-04-18 22:56:13,617 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/a73bf9fe2ea34f4d86c40f120399f70b, entries=150, sequenceid=359, filesize=11.9 K 2023-04-18 22:56:13,617 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for 8181c42622934944fa97cf06c7004a78 in 860ms, sequenceid=359, compaction requested=true 2023-04-18 22:56:13,618 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:13,618 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:13,618 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:13,618 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:13,618 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,618 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:13,618 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:13,623 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,623 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:13,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:13,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:13,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:13,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:13,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3639b5ba[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:13,626 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:13,626 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:13,628 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:13,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:13,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:13,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:13,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:13,628 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:13,628 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,628 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:13,630 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,630 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:13,630 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:13,630 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:13,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:13,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:13,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:13,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:13,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:13,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:13,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:13,634 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:13,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:13,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:13,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:13,634 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:13,635 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,636 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:13,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,635 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:13,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:13,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:13,637 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:13,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:13,638 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:13,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:13,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:13,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:13,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:13,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:13,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:13,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:13,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:13,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:13,644 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:13,644 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:13,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:13,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:13,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:13,645 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:13,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:13,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:13,745 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:13,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:13,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:13,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:13,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:13,750 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:13,750 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:13,750 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:13,750 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:13,751 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,751 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:13,752 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,752 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,752 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:13,752 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:13,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:13,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:13,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:13,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:13,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:13,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:13,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:13,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:13,754 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:13,754 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:13,754 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:13,754 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:13,754 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:13,754 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:13,754 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:13,754 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-18 22:56:13,755 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:13,755 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:13,755 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:13,755 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:13,755 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:13,755 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:13,775 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304186fe1f5b9ca5942989171bb65493de8c6 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186fe1f5b9ca5942989171bb65493de8c6 2023-04-18 22:56:13,786 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=367, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/b0069692dcdd4c909cbe0704ddddba62 2023-04-18 22:56:13,799 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/5936017b764846d5adb005e49f7354fe 2023-04-18 22:56:13,812 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/b8bfd6f8c0834b51b734da8407f552d5 2023-04-18 22:56:13,815 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/b0069692dcdd4c909cbe0704ddddba62 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/b0069692dcdd4c909cbe0704ddddba62 2023-04-18 22:56:13,820 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/b0069692dcdd4c909cbe0704ddddba62, entries=150, sequenceid=367, filesize=25.4 K 2023-04-18 22:56:13,820 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/5936017b764846d5adb005e49f7354fe as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/5936017b764846d5adb005e49f7354fe 2023-04-18 22:56:13,824 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/5936017b764846d5adb005e49f7354fe, entries=150, sequenceid=367, filesize=11.9 K 2023-04-18 22:56:13,824 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/b8bfd6f8c0834b51b734da8407f552d5 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/b8bfd6f8c0834b51b734da8407f552d5 2023-04-18 22:56:13,827 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/b8bfd6f8c0834b51b734da8407f552d5, entries=150, sequenceid=367, filesize=11.9 K 2023-04-18 22:56:13,828 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 8181c42622934944fa97cf06c7004a78 in 74ms, sequenceid=367, compaction requested=true 2023-04-18 22:56:13,828 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:13,828 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:13,828 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:13,828 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:13,828 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,828 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:13,828 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:13,830 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,830 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:13,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:13,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:13,830 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:13,831 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:13,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:13,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:13,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:13,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:13,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:13,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@715b215b[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:13,832 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:13,832 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:13,834 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:13,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:13,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:13,834 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:13,834 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:13,834 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:13,834 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,834 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:13,836 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,836 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:13,836 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:13,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:13,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:13,836 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:13,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:13,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:13,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:13,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:13,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:13,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:13,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:13,845 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:13,845 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:13,845 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:13,845 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:13,845 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:13,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:13,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:13,846 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:13,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:13,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:13,847 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:13,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:13,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:13,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:13,848 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:13,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:13,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,851 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,851 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:13,851 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:13,851 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:13,851 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:13,851 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:13,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:13,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:13,851 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,851 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:13,851 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:13,851 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,851 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:13,851 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:13,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:13,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:13,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:13,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:13,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:13,952 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:13,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:13,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:13,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:13,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:13,956 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:13,956 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:13,956 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:13,956 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:13,956 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,956 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:13,958 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:13,958 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:13,958 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:13,958 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:13,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:13,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:13,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:13,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:13,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:13,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:13,960 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:13,960 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:13,960 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:13,966 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:13,966 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:13,967 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:13,966 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:13,967 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-18 22:56:13,967 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:13,967 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:13,967 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:13,967 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:13,967 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:13,967 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:13,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:13,999 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:14,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858574024, max timestamp=1681858574048Num uniques 859; 2023-04-18 22:56:14,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:56:14,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858574024, max timestamp=1681858574048Num uniques 859; 2023-04-18 22:56:14,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:56:14,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858574024, max timestamp=1681858574048Num uniques 859; 2023-04-18 22:56:14,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:56:14,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858574072, max timestamp=1681858574115Num uniques 1018; 2023-04-18 22:56:14,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-18 22:56:14,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858574072, max timestamp=1681858574115Num uniques 1018; 2023-04-18 22:56:14,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681858574072, max timestamp=1681858574115Num uniques 1018; 2023-04-18 22:56:14,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-18 22:56:14,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-18 22:56:14,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681858574162, max timestamp=1681858574191Num uniques 1027; 2023-04-18 22:56:14,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681858574162, max timestamp=1681858574191Num uniques 1027; 2023-04-18 22:56:14,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681858574162, max timestamp=1681858574191Num uniques 1027; 2023-04-18 22:56:14,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 22:56:14,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 22:56:14,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-18 22:56:14,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681858574238, max timestamp=1681858574258Num uniques 1036; 2023-04-18 22:56:14,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681858574238, max timestamp=1681858574258Num uniques 1036; 2023-04-18 22:56:14,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 22:56:14,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681858574238, max timestamp=1681858574258Num uniques 1036; 2023-04-18 22:56:14,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 22:56:14,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-18 22:56:14,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681858574288, max timestamp=1681858574318Num uniques 1045; 2023-04-18 22:56:14,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-18 22:56:14,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681858574288, max timestamp=1681858574318Num uniques 1045; 2023-04-18 22:56:14,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681858574288, max timestamp=1681858574318Num uniques 1045; 2023-04-18 22:56:14,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-18 22:56:14,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-18 22:56:14,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681858574360, max timestamp=1681858574381Num uniques 1054; 2023-04-18 22:56:14,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-18 22:56:14,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681858574360, max timestamp=1681858574381Num uniques 1054; 2023-04-18 22:56:14,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-18 22:56:14,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681858574360, max timestamp=1681858574381Num uniques 1054; 2023-04-18 22:56:14,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-18 22:56:14,398 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418f943a707609c47a9a460d88d0ea8ef7c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f943a707609c47a9a460d88d0ea8ef7c 2023-04-18 22:56:14,418 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=372, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/8f4e9f9a6c4245869a5e982a5726cd45 2023-04-18 22:56:14,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681858574394, max timestamp=1681858574417Num uniques 1013; 2023-04-18 22:56:14,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681858574394, max timestamp=1681858574417Num uniques 1013; 2023-04-18 22:56:14,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-18 22:56:14,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681858574394, max timestamp=1681858574417Num uniques 1013; 2023-04-18 22:56:14,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-18 22:56:14,420 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-18 22:56:14,426 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/25f04553152d44a59dddc007cf9ac801 2023-04-18 22:56:14,435 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/d926a0d0d06e499a9c479e92c348b1ea 2023-04-18 22:56:14,438 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/8f4e9f9a6c4245869a5e982a5726cd45 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/8f4e9f9a6c4245869a5e982a5726cd45 2023-04-18 22:56:14,441 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/8f4e9f9a6c4245869a5e982a5726cd45, entries=100, sequenceid=372, filesize=18.6 K 2023-04-18 22:56:14,442 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/25f04553152d44a59dddc007cf9ac801 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/25f04553152d44a59dddc007cf9ac801 2023-04-18 22:56:14,444 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/25f04553152d44a59dddc007cf9ac801, entries=50, sequenceid=372, filesize=7.1 K 2023-04-18 22:56:14,445 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/d926a0d0d06e499a9c479e92c348b1ea as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/d926a0d0d06e499a9c479e92c348b1ea 2023-04-18 22:56:14,447 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/d926a0d0d06e499a9c479e92c348b1ea, entries=50, sequenceid=372, filesize=7.1 K 2023-04-18 22:56:14,460 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=107.34 KB/109920 for 8181c42622934944fa97cf06c7004a78 in 493ms, sequenceid=372, compaction requested=true 2023-04-18 22:56:14,461 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:14,461 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:14,461 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:14,461 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:14,461 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,461 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:14,461 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:14,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:14,461 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=120.76 KB heapSize=340.08 KB 2023-04-18 22:56:14,475 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:14,475 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:14,475 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:14,475 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:14,475 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:14,475 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:14,476 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,477 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:14,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:14,477 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,477 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:14,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:14,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:14,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:14,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:14,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@664da5aa[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:14,479 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:14,479 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,480 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:14,481 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,481 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:14,482 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:14,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:14,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:14,482 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:14,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:14,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:14,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:14,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:14,486 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:14,486 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:14,486 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:14,486 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:14,486 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:14,488 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,488 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:14,488 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:14,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:14,489 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:14,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:14,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:14,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858634484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:14,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858634485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:14,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:14,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,491 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858634486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858634485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,492 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041889c78ffffb34445997fee42d545e37e4 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041889c78ffffb34445997fee42d545e37e4 2023-04-18 22:56:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:14,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:14,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:14,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:14,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,495 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:14,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,495 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:14,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:14,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:14,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:14,495 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:14,496 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:14,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:14,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858634491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:14,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858634592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:14,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858634592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:14,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858634592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:14,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858634593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,596 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:14,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:14,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:14,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:14,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858634599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:14,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:14,599 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:14,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:14,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:14,599 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:14,600 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,600 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,601 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:14,601 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:14,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:14,601 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,601 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:14,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:14,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:14,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:14,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:14,602 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:14,602 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:14,602 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:14,604 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:14,604 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:14,604 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:14,604 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:14,604 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:14,606 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:14,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,606 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:14,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:14,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:14,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:14,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:14,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d51493a[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:14,608 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:14,608 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,617 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,617 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:14,617 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:14,617 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,617 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:14,617 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:14,619 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:14,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:14,619 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:14,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:14,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:14,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:14,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:14,621 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:14,621 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:14,621 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:14,621 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:14,621 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:14,623 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,623 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,623 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:14,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:14,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:14,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:14,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,624 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,625 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:14,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,627 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,627 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,627 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,627 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:14,627 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:14,627 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,627 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:14,627 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:14,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:14,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:14,627 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:14,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:14,628 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:14,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:14,628 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:14,628 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:14,728 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:14,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:14,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:14,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:14,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:14,733 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:14,733 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:14,733 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:14,733 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:14,734 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,734 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,736 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,736 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:14,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:14,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:14,736 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,736 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:14,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:14,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:14,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:14,738 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:14,738 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:14,738 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:14,738 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:14,738 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:14,738 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:14,738 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:14,738 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:14,738 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:14,738 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:14,738 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:14,738 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,738 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:14,738 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:14,739 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,740 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:14,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:14,740 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,740 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:14,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:14,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:14,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:14,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:14,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@763ea94f[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:14,742 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:14,742 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,744 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,744 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:14,744 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:14,744 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:14,744 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,744 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:14,745 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,745 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:14,745 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:14,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:14,746 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:14,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:14,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:14,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:14,751 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:14,751 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:14,751 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:14,751 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:14,751 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:14,753 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,753 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:14,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:14,753 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:14,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:14,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:14,754 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,755 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:14,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,757 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,757 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,757 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,758 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,758 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:14,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:14,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:14,758 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,758 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:14,758 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,758 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:14,758 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:14,758 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:14,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:14,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:14,758 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:14,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:14,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:14,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858634797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:14,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858634797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:14,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858634797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:14,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858634799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:14,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858634801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,858 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:14,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:14,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:14,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:14,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:14,861 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:14,862 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:14,862 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:14,862 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:14,862 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,862 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,863 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:14,863 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:14,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:14,864 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,864 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:14,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:14,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:14,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:14,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:14,865 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:14,865 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:14,865 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:14,865 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:14,865 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:14,865 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:14,865 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:14,865 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:14,865 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:14,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:14,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:14,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:14,866 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:14,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,867 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:14,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,867 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:14,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:14,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:14,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ec70f2a[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:14,869 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:14,869 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,871 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,871 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:14,871 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:14,871 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:14,871 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,871 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:14,872 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,872 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:14,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:14,872 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:14,873 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:14,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:14,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:14,875 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:14,875 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:14,875 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:14,875 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:14,875 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:14,876 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,877 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,877 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:14,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:14,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:14,877 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,878 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:14,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:14,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:14,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:14,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:14,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:14,881 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,881 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:14,881 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:14,881 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:14,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:14,900 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=511, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/4fd3eb43719941fb950c8671bb06f48d 2023-04-18 22:56:14,982 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:14,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:14,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:14,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:14,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:14,985 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:14,985 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:14,985 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:14,985 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:14,986 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,986 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,988 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:14,988 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:14,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:14,989 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,989 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:14,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:14,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:14,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:14,990 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:14,990 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:14,990 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:14,990 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:14,990 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:14,990 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:14,990 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:14,991 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:14,991 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:14,991 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:14,991 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:14,991 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,991 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:14,991 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:14,993 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,993 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:14,993 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:14,993 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:14,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:14,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:14,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55ac81ad[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:14,995 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:14,995 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,996 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:14,997 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:14,997 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:14,997 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:14,997 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,997 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:14,998 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,998 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:14,998 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:14,998 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:15,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:15,001 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:15,002 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:15,002 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:15,002 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:15,002 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:15,007 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,008 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:15,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:15,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,008 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:15,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:15,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:15,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:15,008 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:15,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:15,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,011 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,015 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:15,015 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:15,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:15,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:15,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:15,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:15,016 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,016 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,016 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,016 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,016 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,016 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,016 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,016 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:15,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:15,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:15,016 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:15,016 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:15,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:15,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858635101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:15,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858635102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:15,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858635103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:15,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858635105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:15,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858635108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,116 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:15,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:15,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:15,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:15,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:15,120 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:15,120 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:15,120 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:15,120 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:15,121 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,121 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,122 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:15,122 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,123 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,123 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:15,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:15,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:15,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:15,124 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:15,124 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:15,124 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:15,124 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:15,124 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:15,126 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,126 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:15,126 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,126 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:15,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:15,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:15,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:15,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52b31abb[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:15,129 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:15,129 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,130 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,130 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:15,130 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:15,130 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:15,131 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,131 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:15,137 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:15,137 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:15,137 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,137 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:15,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:15,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:15,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:15,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:15,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:15,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:15,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:15,140 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:15,140 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:15,140 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:15,140 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:15,140 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:15,142 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,142 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:15,142 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,142 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:15,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:15,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,144 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:15,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,146 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,146 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:15,146 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:15,146 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:15,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:15,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:15,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:15,147 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:15,146 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:15,147 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:15,146 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,147 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:15,147 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,147 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:15,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:15,147 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,247 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:15,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:15,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:15,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:15,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:15,251 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:15,251 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:15,251 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:15,251 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:15,251 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,251 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,253 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:15,253 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:15,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:15,253 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,253 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:15,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:15,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:15,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:15,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:15,255 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:15,255 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:15,255 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:15,255 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:15,255 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:15,255 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:15,255 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:15,255 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:15,255 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:15,255 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:15,255 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:15,255 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,255 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:15,255 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:15,257 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,257 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:15,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,258 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:15,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:15,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5144a573[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:15,259 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:15,259 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,260 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,260 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:15,260 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:15,260 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:15,260 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,260 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:15,262 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:15,262 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,262 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:15,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:15,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:15,262 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:15,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:15,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:15,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:15,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:15,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:15,264 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:15,264 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:15,264 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:15,264 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:15,264 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:15,266 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,266 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:15,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:15,266 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:15,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:15,266 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:15,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,268 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:15,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:15,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,272 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:15,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:15,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:15,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:15,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:15,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:15,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:15,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,272 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:15,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:15,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:15,272 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:15,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,272 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,310 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=511 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/d497ce3e340140e88b0a1f3871c3e569 2023-04-18 22:56:15,372 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:15,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:15,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:15,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:15,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:15,376 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:15,376 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:15,376 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:15,376 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:15,377 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,377 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,378 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:15,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:15,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:15,378 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,379 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,379 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:15,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:15,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:15,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:15,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:15,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:15,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:15,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:15,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:15,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:15,380 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:15,380 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:15,380 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:15,380 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:15,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:15,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:15,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:15,380 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:15,382 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,382 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:15,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:15,382 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,382 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:15,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:15,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:15,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:15,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:15,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@371c4e2e[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:15,383 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:15,384 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,384 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:15,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:15,385 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:15,385 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,385 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:15,387 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,387 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:15,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:15,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:15,387 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:15,387 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:15,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:15,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:15,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:15,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:15,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:15,390 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:15,390 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:15,390 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:15,390 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:15,390 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:15,392 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,392 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:15,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:15,392 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:15,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:15,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:15,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:15,393 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:15,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,395 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:15,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,400 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,400 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,400 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,400 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,401 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,401 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,401 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,401 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:15,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:15,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:15,401 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:15,401 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:15,401 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:15,401 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:15,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:15,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:15,501 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:15,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:15,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:15,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:15,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:15,505 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:15,505 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:15,505 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:15,505 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:15,505 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,506 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,508 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:15,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:15,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:15,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:15,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:15,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:15,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:15,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:15,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:15,510 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:15,513 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:15,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:15,513 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:15,513 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:15,513 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:15,513 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:15,513 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:15,514 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,514 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:15,514 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:15,514 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,514 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:15,514 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:15,515 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,515 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:15,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:15,516 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,516 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:15,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:15,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:15,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:15,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:15,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60c1089[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:15,517 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:15,517 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,519 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,519 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:15,519 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:15,519 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:15,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,519 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:15,520 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,520 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:15,521 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:15,521 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:15,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:15,523 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:15,523 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:15,523 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:15,523 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:15,523 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:15,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:15,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,525 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:15,526 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,527 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:15,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,529 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:15,529 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:15,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:15,529 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:15,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:15,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:15,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:15,529 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:15,530 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,530 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:15,530 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,530 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:15,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:15,530 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,530 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,530 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,530 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:15,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858635606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:15,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858635606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:15,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858635607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:15,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858635610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:15,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858635612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,630 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:15,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:15,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:15,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:15,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:15,633 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:15,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:15,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:15,633 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:15,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,634 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,635 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:15,635 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:15,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:15,636 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,636 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:15,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:15,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:15,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:15,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:15,637 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:15,637 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:15,637 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:15,637 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:15,637 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:15,637 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:15,637 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:15,637 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:15,637 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:15,637 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:15,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:15,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:15,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:15,639 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,639 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:15,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:15,640 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,640 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:15,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:15,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:15,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:15,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:15,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c20b431[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:15,642 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:15,642 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,644 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,644 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:15,644 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:15,644 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:15,644 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,644 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:15,645 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:15,645 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,645 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:15,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,645 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:15,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:15,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:15,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:15,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:15,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:15,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:15,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:15,649 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:15,649 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:15,649 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:15,649 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:15,649 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:15,651 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,651 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,651 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:15,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:15,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:15,652 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:15,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:15,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:15,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:15,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,653 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:15,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:15,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,656 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,656 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,656 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,656 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,656 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:15,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:15,656 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,657 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:15,657 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:15,656 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:15,657 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:15,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:15,657 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:15,657 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:15,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:15,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:15,721 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=511 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/048a250d600445abb38f5c6a452cd329 2023-04-18 22:56:15,725 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/4fd3eb43719941fb950c8671bb06f48d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4fd3eb43719941fb950c8671bb06f48d 2023-04-18 22:56:15,729 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4fd3eb43719941fb950c8671bb06f48d, entries=250, sequenceid=511, filesize=39.2 K 2023-04-18 22:56:15,730 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/d497ce3e340140e88b0a1f3871c3e569 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/d497ce3e340140e88b0a1f3871c3e569 2023-04-18 22:56:15,733 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/d497ce3e340140e88b0a1f3871c3e569, entries=150, sequenceid=511, filesize=11.9 K 2023-04-18 22:56:15,734 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/048a250d600445abb38f5c6a452cd329 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/048a250d600445abb38f5c6a452cd329 2023-04-18 22:56:15,737 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/048a250d600445abb38f5c6a452cd329, entries=150, sequenceid=511, filesize=11.9 K 2023-04-18 22:56:15,738 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.88 KB/388992, currentSize=46.96 KB/48090 for 8181c42622934944fa97cf06c7004a78 in 1277ms, sequenceid=511, compaction requested=true 2023-04-18 22:56:15,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:15,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:15,738 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-18 22:56:15,738 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-18 22:56:15,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:15,739 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:15,752 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 177906 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-18 22:56:15,752 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 68742 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-18 22:56:15,753 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/A is initiating minor compaction (all files) 2023-04-18 22:56:15,753 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/A in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:15,753 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4ee4ba3abe8b402dbd123d83388d7c60, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/9ecd515dae9e48ca8abdf72718dc331f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/ad9466a783bd4113b5f94da704bb25d5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/b0069692dcdd4c909cbe0704ddddba62, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/8f4e9f9a6c4245869a5e982a5726cd45, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4fd3eb43719941fb950c8671bb06f48d] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=173.7 K 2023-04-18 22:56:15,753 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/B is initiating minor compaction (all files) 2023-04-18 22:56:15,753 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/B in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:15,753 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 4ee4ba3abe8b402dbd123d83388d7c60, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1681858572121 2023-04-18 22:56:15,753 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f581499d082d4841bf4fefc4da276d0e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/e5db8fe6d5f74b73878b54af650f6257, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/88a612595ce84c21b710bef84803aea2, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/5936017b764846d5adb005e49f7354fe, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/25f04553152d44a59dddc007cf9ac801, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/d497ce3e340140e88b0a1f3871c3e569] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=67.1 K 2023-04-18 22:56:15,754 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 9ecd515dae9e48ca8abdf72718dc331f, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1681858572186 2023-04-18 22:56:15,754 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting f581499d082d4841bf4fefc4da276d0e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1681858572121 2023-04-18 22:56:15,754 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting ad9466a783bd4113b5f94da704bb25d5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1681858572252 2023-04-18 22:56:15,754 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting e5db8fe6d5f74b73878b54af650f6257, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1681858572186 2023-04-18 22:56:15,754 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting b0069692dcdd4c909cbe0704ddddba62, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681858572807 2023-04-18 22:56:15,755 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 88a612595ce84c21b710bef84803aea2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1681858572252 2023-04-18 22:56:15,755 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 8f4e9f9a6c4245869a5e982a5726cd45, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681858573949 2023-04-18 22:56:15,755 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 5936017b764846d5adb005e49f7354fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681858572807 2023-04-18 22:56:15,755 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 4fd3eb43719941fb950c8671bb06f48d, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1681858574432 2023-04-18 22:56:15,755 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 25f04553152d44a59dddc007cf9ac801, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681858573965 2023-04-18 22:56:15,756 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting d497ce3e340140e88b0a1f3871c3e569, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1681858574437 2023-04-18 22:56:15,757 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:15,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:15,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:15,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:15,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:15,761 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:15,761 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:15,761 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:15,762 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:15,762 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,762 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,765 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:15,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:15,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:15,765 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,765 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:15,765 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:15,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:15,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:15,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:15,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:15,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:15,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:15,768 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:15,768 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:15,768 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:15,768 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:15,768 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:15,768 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:15,768 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:15,769 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:56:15,769 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:15,769 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:15,769 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:15,769 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:15,769 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:15,769 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:15,770 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#B#compaction#551 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:15,772 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#A#compaction#550 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:16,205 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/42d62044040149f3a86b548d01e2226f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/42d62044040149f3a86b548d01e2226f 2023-04-18 22:56:16,210 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041841853b27e7a24581b77e6361d07531a9 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041841853b27e7a24581b77e6361d07531a9 2023-04-18 22:56:16,211 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 8181c42622934944fa97cf06c7004a78/B of 8181c42622934944fa97cf06c7004a78 into 42d62044040149f3a86b548d01e2226f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:16,211 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:16,211 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/B, priority=10, startTime=1681858575738; duration=0sec 2023-04-18 22:56:16,211 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:16,211 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-18 22:56:16,218 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/d4201c055ec5465a9eeda5bbcca1f555 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/d4201c055ec5465a9eeda5bbcca1f555 2023-04-18 22:56:16,218 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 68742 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-18 22:56:16,218 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/C is initiating minor compaction (all files) 2023-04-18 22:56:16,219 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/C in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:16,219 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/49d5072797b84e61b9178384f0e33263, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/43e7a9ffc77641dab32944f4f2a2b324, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/a73bf9fe2ea34f4d86c40f120399f70b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/b8bfd6f8c0834b51b734da8407f552d5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/d926a0d0d06e499a9c479e92c348b1ea, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/048a250d600445abb38f5c6a452cd329] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=67.1 K 2023-04-18 22:56:16,219 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 49d5072797b84e61b9178384f0e33263, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1681858572121 2023-04-18 22:56:16,220 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 43e7a9ffc77641dab32944f4f2a2b324, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1681858572186 2023-04-18 22:56:16,220 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting a73bf9fe2ea34f4d86c40f120399f70b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1681858572252 2023-04-18 22:56:16,220 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting b8bfd6f8c0834b51b734da8407f552d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681858572807 2023-04-18 22:56:16,221 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting d926a0d0d06e499a9c479e92c348b1ea, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681858573965 2023-04-18 22:56:16,221 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 048a250d600445abb38f5c6a452cd329, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1681858574437 2023-04-18 22:56:16,223 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 8181c42622934944fa97cf06c7004a78/A of 8181c42622934944fa97cf06c7004a78 into d4201c055ec5465a9eeda5bbcca1f555(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:16,223 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:16,223 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/A, priority=10, startTime=1681858575738; duration=0sec 2023-04-18 22:56:16,223 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:16,237 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#C#compaction#553 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:16,250 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/e9756b57bb964725a38da3d5ce2067b5 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/e9756b57bb964725a38da3d5ce2067b5 2023-04-18 22:56:16,254 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 8181c42622934944fa97cf06c7004a78/C of 8181c42622934944fa97cf06c7004a78 into e9756b57bb964725a38da3d5ce2067b5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:16,254 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:16,254 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/C, priority=10, startTime=1681858575739; duration=0sec 2023-04-18 22:56:16,254 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:16,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:16,613 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:16,621 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=521, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/5efc5efe80dd4f228207b7734bf85939 2023-04-18 22:56:16,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858576626, max timestamp=1681858576657Num uniques 859; 2023-04-18 22:56:16,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:56:16,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858576626, max timestamp=1681858576657Num uniques 859; 2023-04-18 22:56:16,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:56:16,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858576626, max timestamp=1681858576657Num uniques 859; 2023-04-18 22:56:16,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:56:16,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:16,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858636668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:16,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:16,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858636669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:16,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:16,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858636669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:16,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:16,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858636670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:16,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:16,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858636670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:16,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:16,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858636772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:16,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:16,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858636773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:16,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:16,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858636773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:16,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:16,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858636774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:16,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:16,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858636774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:16,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:16,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858636974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:16,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:16,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858636974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:16,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:16,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858636977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:16,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:16,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858636977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:16,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:16,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858636978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,037 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/e4015a6e8a1c46d5b5c385c73017bf2e 2023-04-18 22:56:17,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:17,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45800 deadline: 1681858637278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:17,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45876 deadline: 1681858637278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:17,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:45846 deadline: 1681858637280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:17,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858637281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:17,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858637281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,447 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/dca728a880eb4ba5b3924447c992b3f2 2023-04-18 22:56:17,454 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/5efc5efe80dd4f228207b7734bf85939 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/5efc5efe80dd4f228207b7734bf85939 2023-04-18 22:56:17,457 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/5efc5efe80dd4f228207b7734bf85939, entries=100, sequenceid=521, filesize=18.6 K 2023-04-18 22:56:17,458 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/e4015a6e8a1c46d5b5c385c73017bf2e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/e4015a6e8a1c46d5b5c385c73017bf2e 2023-04-18 22:56:17,461 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/e4015a6e8a1c46d5b5c385c73017bf2e, entries=100, sequenceid=521, filesize=9.5 K 2023-04-18 22:56:17,462 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/dca728a880eb4ba5b3924447c992b3f2 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/dca728a880eb4ba5b3924447c992b3f2 2023-04-18 22:56:17,465 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/dca728a880eb4ba5b3924447c992b3f2, entries=100, sequenceid=521, filesize=9.5 K 2023-04-18 22:56:17,465 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 8181c42622934944fa97cf06c7004a78 in 1696ms, sequenceid=521, compaction requested=false 2023-04-18 22:56:17,466 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:17,466 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:17,466 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:17,466 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:17,466 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,466 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:17,466 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:17,467 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,467 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:17,468 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:17,468 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:17,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:17,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:17,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@383beffb[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:17,470 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:17,470 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:17,471 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:17,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:17,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:17,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:17,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:17,471 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:17,471 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,471 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:17,473 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,473 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:17,473 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:17,473 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:17,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:17,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:17,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:17,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:17,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:17,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:17,476 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:17,476 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:17,476 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:17,476 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:17,476 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:17,483 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:17,483 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:17,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:17,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:17,483 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:17,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:17,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:17,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:17,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:17,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:17,483 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:17,485 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:17,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,489 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,489 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:17,489 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:17,489 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:17,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:17,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:17,489 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:17,489 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,489 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:17,490 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:17,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:17,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:17,489 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:17,489 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:17,490 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:17,490 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:17,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:17,590 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:17,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:17,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:17,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:17,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:17,593 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:17,593 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:17,593 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:17,593 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:17,594 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:17,594 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:17,595 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:17,595 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:17,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:17,596 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,596 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:17,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:17,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:17,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:17,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:17,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:17,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:17,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:17,597 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:17,597 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:17,597 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:17,597 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:17,597 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:17,597 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:17,597 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:17,598 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-18 22:56:17,598 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:17,598 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:17,598 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:17,598 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:17,598 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:17,598 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:17,611 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418e09e56dc2178498d91b6d1f7ddbb611d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e09e56dc2178498d91b6d1f7ddbb611d 2023-04-18 22:56:17,616 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=561, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/3c1de124c9b9479a8f4e11c28ac7bf33 2023-04-18 22:56:17,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:17,785 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:17,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:17,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858637787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:17,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858637788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:17,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:17,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858637789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858637789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:17,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858637790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:17,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858637893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:17,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858637893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:17,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858637893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:17,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:17,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858637893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,027 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/f970ec7d19d44f788760be726752346b 2023-04-18 22:56:18,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:18,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858638095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:18,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:18,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858638095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858638096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:18,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858638096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:18,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858638400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:18,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858638400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:18,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858638400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:18,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858638401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,438 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/1eae5d3a839c45ca88a6cbe3045cba0a 2023-04-18 22:56:18,442 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/3c1de124c9b9479a8f4e11c28ac7bf33 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/3c1de124c9b9479a8f4e11c28ac7bf33 2023-04-18 22:56:18,445 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/3c1de124c9b9479a8f4e11c28ac7bf33, entries=150, sequenceid=561, filesize=25.4 K 2023-04-18 22:56:18,445 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/f970ec7d19d44f788760be726752346b as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f970ec7d19d44f788760be726752346b 2023-04-18 22:56:18,448 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f970ec7d19d44f788760be726752346b, entries=150, sequenceid=561, filesize=11.9 K 2023-04-18 22:56:18,449 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/1eae5d3a839c45ca88a6cbe3045cba0a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/1eae5d3a839c45ca88a6cbe3045cba0a 2023-04-18 22:56:18,453 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/1eae5d3a839c45ca88a6cbe3045cba0a, entries=150, sequenceid=561, filesize=11.9 K 2023-04-18 22:56:18,454 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 8181c42622934944fa97cf06c7004a78 in 856ms, sequenceid=561, compaction requested=true 2023-04-18 22:56:18,454 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:18,454 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:18,454 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:18,454 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:18,454 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,455 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:18,455 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:18,457 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:18,457 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:18,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:18,457 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,457 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:18,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:18,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d03bbec[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:18,459 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:18,459 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,460 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,460 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:18,460 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:18,460 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,460 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:18,460 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:18,461 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,461 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:18,462 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:18,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,462 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:18,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:18,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:18,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:18,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:18,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:18,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:18,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:18,464 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:18,464 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:18,464 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:18,464 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:18,464 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:18,466 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,466 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:18,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,466 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:18,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:18,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:18,467 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:18,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,468 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:18,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,472 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,472 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:18,472 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:18,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:18,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:18,472 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:18,472 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,472 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,472 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,472 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:18,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:18,472 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:18,472 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:18,472 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:18,472 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:18,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:18,573 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:18,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:18,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:18,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:18,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:18,576 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:18,576 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:18,576 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:18,577 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:18,577 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,577 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:18,578 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,578 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:18,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:18,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:18,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:18,579 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,579 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:18,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:18,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:18,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:18,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:18,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:18,580 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:18,580 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:18,580 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:18,580 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:18,580 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:18,580 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:18,580 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:18,580 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-18 22:56:18,580 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:18,581 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:18,581 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:18,581 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:18,581 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:18,581 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:18,593 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304189e7c96ddbcc9450b9121a9dd9d5c843f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304189e7c96ddbcc9450b9121a9dd9d5c843f 2023-04-18 22:56:18,598 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=570, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/a47d1ce870ae4d0d9b8103e9138698e9 2023-04-18 22:56:18,609 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/c04e572246db4859b25332f9d9ee86d4 2023-04-18 22:56:18,617 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=570 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/b1fde4b3f0f545fe920d558efaf0e310 2023-04-18 22:56:18,621 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/a47d1ce870ae4d0d9b8103e9138698e9 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/a47d1ce870ae4d0d9b8103e9138698e9 2023-04-18 22:56:18,624 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/a47d1ce870ae4d0d9b8103e9138698e9, entries=150, sequenceid=570, filesize=25.4 K 2023-04-18 22:56:18,625 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/c04e572246db4859b25332f9d9ee86d4 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/c04e572246db4859b25332f9d9ee86d4 2023-04-18 22:56:18,628 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/c04e572246db4859b25332f9d9ee86d4, entries=150, sequenceid=570, filesize=11.9 K 2023-04-18 22:56:18,628 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/b1fde4b3f0f545fe920d558efaf0e310 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/b1fde4b3f0f545fe920d558efaf0e310 2023-04-18 22:56:18,632 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/b1fde4b3f0f545fe920d558efaf0e310, entries=150, sequenceid=570, filesize=11.9 K 2023-04-18 22:56:18,633 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 8181c42622934944fa97cf06c7004a78 in 52ms, sequenceid=570, compaction requested=true 2023-04-18 22:56:18,633 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:18,633 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:18,633 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:18,633 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:18,633 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,633 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:18,633 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:18,634 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,634 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:18,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,635 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:18,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:18,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bed669d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:18,637 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:18,637 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,638 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:18,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:18,638 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:18,638 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,638 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:18,654 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:18,654 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:18,654 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:18,654 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:18,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:18,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:18,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:18,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:18,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:18,657 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:18,657 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:18,657 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:18,657 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:18,657 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:18,659 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,659 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,659 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:18,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:18,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:18,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:18,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:18,660 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:18,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,661 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:18,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:18,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:18,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:18,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,664 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:18,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:18,663 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:18,663 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:18,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:18,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:18,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:18,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:18,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:18,664 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:18,664 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:18,765 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:18,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:18,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:18,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:18,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:18,769 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:18,769 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:18,769 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:18,769 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:18,769 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,769 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:18,771 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:18,771 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:18,771 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,771 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:18,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:18,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:18,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:18,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:18,772 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:18,772 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:18,772 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:18,773 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:18,773 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:18,773 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:18,774 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:18,775 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:18,775 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:18,775 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:18,775 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:18,775 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,775 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:18,775 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:18,778 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,778 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:18,778 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,778 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:18,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:18,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:18,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4225daed[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:18,780 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:18,780 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,781 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,781 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:18,781 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:18,781 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:18,781 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,781 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:18,782 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,782 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:18,782 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:18,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:18,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:18,782 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:18,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:18,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:18,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:18,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:18,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:18,785 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:18,785 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:18,785 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:18,785 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:18,785 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:18,787 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,787 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:18,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:18,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,787 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:18,788 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:18,789 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:18,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,794 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,794 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:18,794 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:18,794 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,794 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,794 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,794 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,794 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:18,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:18,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:18,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:18,794 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:18,795 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:18,795 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:18,795 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:18,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:18,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:18,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:18,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:18,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:18,853 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:56:18,854 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:18,854 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:18,854 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:18,854 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:18,854 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:18,854 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:18,871 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304185c0e5fd3d8b54e5aa1b08e25929731e3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304185c0e5fd3d8b54e5aa1b08e25929731e3 2023-04-18 22:56:18,895 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:18,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:18,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:18,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:18,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:18,898 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:18,898 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:18,898 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:18,898 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:18,898 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,898 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:18,899 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:18,899 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:18,900 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,900 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:18,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:18,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:18,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:18,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:18,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:18,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:18,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:18,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:18,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:18,901 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:18,901 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:18,901 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:18,901 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:18,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:18,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:18,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:18,901 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:18,905 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,905 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:18,906 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,906 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:18,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:18,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:18,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2586a165[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:18,908 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:18,908 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,910 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:18,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:18,910 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:18,910 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,910 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:18,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:18,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858578891, max timestamp=1681858578911Num uniques 859; 2023-04-18 22:56:18,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858578891, max timestamp=1681858578911Num uniques 859; 2023-04-18 22:56:18,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:56:18,913 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:18,913 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:18,913 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:18,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:56:18,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681858578891, max timestamp=1681858578911Num uniques 859; 2023-04-18 22:56:18,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858638909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-18 22:56:18,913 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:18,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:18,916 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:18,917 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:18,917 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:18,917 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:18,917 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:18,918 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,918 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,918 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:18,919 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:18,920 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:18,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:18,922 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:18,922 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:18,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:18,922 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:18,922 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:18,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:18,922 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:18,922 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:18,922 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:18,923 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:18,923 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:18,923 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,923 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:18,923 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:18,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:18,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:18,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858638925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:18,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858638927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:18,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858638927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:18,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:18,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858638929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:19,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858639014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,023 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:19,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:19,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:19,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:19,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:19,026 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:19,026 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:19,027 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:19,027 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:19,027 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,027 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,030 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:19,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:19,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:19,030 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,030 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,030 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:19,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:19,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:19,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:19,031 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:19,031 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:19,031 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:19,031 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:19,032 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:19,032 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:19,032 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:19,032 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:19,032 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:19,032 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:19,032 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:19,032 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,032 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:19,032 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:19,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:19,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858639030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,034 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,034 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:19,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:19,034 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,034 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:19,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:19,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:19,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:19,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:19,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858639031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858639031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:19,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:19,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@552ab2e2[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:19,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:19,036 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:19,037 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858639033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,038 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,038 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:19,038 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:19,038 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:19,038 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,039 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:19,040 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:19,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:19,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:19,043 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:19,043 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:19,043 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:19,043 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:19,043 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:19,045 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,046 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:19,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,046 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:19,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:19,047 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,048 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,050 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,050 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,050 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,050 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,050 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,050 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:19,050 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:19,050 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,050 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:19,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:19,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:19,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:19,050 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:19,050 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:19,051 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:19,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:19,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:19,151 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:19,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:19,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:19,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:19,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:19,154 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:19,155 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:19,155 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:19,155 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:19,155 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,155 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,156 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:19,157 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:19,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:19,157 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,157 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:19,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:19,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:19,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:19,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:19,158 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:19,158 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:19,158 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:19,158 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:19,158 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:19,158 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:19,159 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:19,159 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:19,159 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:19,161 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,161 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:19,161 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,161 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:19,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:19,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:19,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:19,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:19,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3593ff54[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:19,163 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:19,163 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,165 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,165 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:19,165 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:19,165 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:19,167 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:19,167 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:19,167 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:19,167 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:19,167 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:19,167 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:19,167 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:19,167 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:19,169 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,169 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,169 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:19,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,170 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:19,171 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,174 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,174 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,174 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:19,174 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:19,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:19,174 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:19,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:19,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:19,174 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,175 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,175 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,175 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,174 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:19,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:19,175 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:19,175 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:19,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:19,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858639219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:19,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858639233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:19,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858639236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:19,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858639238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:19,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858639238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,275 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:19,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:19,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:19,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:19,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:19,279 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:19,279 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:19,279 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:19,279 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:19,279 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,279 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,283 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:19,283 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:19,283 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=581, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/20e31f018d21473fb7ccc72315b0f701 2023-04-18 22:56:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,284 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,284 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:19,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:19,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:19,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:19,285 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:19,285 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:19,285 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:19,285 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:19,285 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:19,285 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:19,285 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:19,285 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:19,285 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:19,288 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:19,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:19,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:19,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:19,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,290 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:19,290 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,290 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:19,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:19,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@590da9b9[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:19,290 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:19,290 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,294 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,294 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:19,294 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:19,294 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,294 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:19,294 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:19,297 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:19,297 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:19,297 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:19,297 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:19,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:19,300 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:19,300 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:19,300 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:19,300 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:19,300 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:19,301 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/76174215266648d38f78e27802e3b662 2023-04-18 22:56:19,302 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,302 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:19,302 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:19,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:19,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,303 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:19,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:19,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,305 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,308 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,308 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,308 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,308 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:19,308 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:19,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:19,308 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,309 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,309 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,309 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:19,309 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:19,309 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:19,309 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:19,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:19,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:19,409 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:19,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:19,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:19,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:19,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:19,413 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:19,413 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:19,413 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:19,413 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:19,413 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,413 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,415 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,415 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:19,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:19,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:19,415 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,415 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:19,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:19,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:19,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:19,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:19,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:19,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:19,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:19,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:19,417 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:19,417 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:19,417 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:19,417 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:19,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:19,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:19,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:19,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:19,423 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,423 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:19,424 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,424 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:19,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:19,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4910b674[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:19,425 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:19,426 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,427 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,427 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:19,427 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:19,427 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:19,427 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,427 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:19,428 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:19,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:19,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:19,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:19,432 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:19,432 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:19,432 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:19,432 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:19,432 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:19,433 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,434 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:19,434 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:19,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:19,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:19,434 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:19,435 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,439 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,439 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:19,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:19,439 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,440 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,440 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,440 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,440 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,439 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:19,440 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:19,440 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:19,440 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:19,440 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:19,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:19,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:19,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:19,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858639524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:19,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858639538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,540 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:19,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:19,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:19,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:19,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:19,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858639541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:19,544 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:19,544 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:19,544 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:19,544 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:19,544 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,544 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:19,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858639544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,546 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:19,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:19,546 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:19,546 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,546 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:19,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:19,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858639545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:19,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:19,548 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:19,548 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:19,548 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:19,548 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:19,548 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:19,548 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:19,548 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:19,548 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:19,548 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:19,548 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:19,548 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:19,548 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,548 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:19,548 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:19,550 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,550 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:19,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:19,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:19,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:19,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:19,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:19,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:19,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@492da016[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:19,553 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:19,553 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,554 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:19,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:19,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:19,554 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,555 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:19,557 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:19,557 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,557 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:19,558 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:19,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:19,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:19,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:19,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:19,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:19,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:19,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:19,560 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:19,560 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:19,560 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:19,560 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:19,560 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:19,562 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,562 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,562 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:19,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:19,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:19,562 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:19,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,564 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:19,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,567 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:19,567 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:19,567 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:19,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:19,567 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,567 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:19,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:19,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:19,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:19,567 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,568 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:19,568 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:19,568 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,568 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,568 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,568 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:19,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:19,668 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:19,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:19,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:19,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:19,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:19,671 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:19,672 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:19,672 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:19,672 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:19,672 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,672 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,674 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:19,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:19,674 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:19,674 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,674 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:19,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:19,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:19,676 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:19,676 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:19,676 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:19,676 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:19,677 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:19,684 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:19,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:19,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@636e8f38[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:19,686 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:19,686 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,687 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,688 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:19,688 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:19,688 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:19,688 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,688 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:19,691 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:19,691 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:19,691 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:19,691 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:19,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:19,694 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:19,694 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:19,694 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:19,694 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:19,694 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:19,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:19,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:19,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:19,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:19,697 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:19,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:19,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,703 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:19,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:19,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:19,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:19,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:19,703 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,704 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:19,704 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:19,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:19,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/25bbed8dd3b44cd989a6ec2175c7ddae 2023-04-18 22:56:19,718 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/20e31f018d21473fb7ccc72315b0f701 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/20e31f018d21473fb7ccc72315b0f701 2023-04-18 22:56:19,722 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/20e31f018d21473fb7ccc72315b0f701, entries=150, sequenceid=581, filesize=25.4 K 2023-04-18 22:56:19,723 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/76174215266648d38f78e27802e3b662 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/76174215266648d38f78e27802e3b662 2023-04-18 22:56:19,726 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/76174215266648d38f78e27802e3b662, entries=150, sequenceid=581, filesize=11.9 K 2023-04-18 22:56:19,727 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/25bbed8dd3b44cd989a6ec2175c7ddae as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/25bbed8dd3b44cd989a6ec2175c7ddae 2023-04-18 22:56:19,730 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/25bbed8dd3b44cd989a6ec2175c7ddae, entries=150, sequenceid=581, filesize=11.9 K 2023-04-18 22:56:19,731 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 8181c42622934944fa97cf06c7004a78 in 878ms, sequenceid=581, compaction requested=true 2023-04-18 22:56:19,731 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:19,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:19,731 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 22:56:19,732 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 22:56:19,732 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:19,732 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:19,733 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 123898 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 22:56:19,733 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/A is initiating minor compaction (all files) 2023-04-18 22:56:19,733 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/A in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:19,733 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/d4201c055ec5465a9eeda5bbcca1f555, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/5efc5efe80dd4f228207b7734bf85939, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/3c1de124c9b9479a8f4e11c28ac7bf33, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/a47d1ce870ae4d0d9b8103e9138698e9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/20e31f018d21473fb7ccc72315b0f701] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=121.0 K 2023-04-18 22:56:19,734 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting d4201c055ec5465a9eeda5bbcca1f555, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1681858574437 2023-04-18 22:56:19,734 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59202 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 22:56:19,734 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/B is initiating minor compaction (all files) 2023-04-18 22:56:19,734 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/B in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:19,734 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/42d62044040149f3a86b548d01e2226f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/e4015a6e8a1c46d5b5c385c73017bf2e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f970ec7d19d44f788760be726752346b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/c04e572246db4859b25332f9d9ee86d4, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/76174215266648d38f78e27802e3b662] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=57.8 K 2023-04-18 22:56:19,734 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 5efc5efe80dd4f228207b7734bf85939, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1681858574484 2023-04-18 22:56:19,735 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 42d62044040149f3a86b548d01e2226f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1681858574437 2023-04-18 22:56:19,735 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 3c1de124c9b9479a8f4e11c28ac7bf33, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1681858576667 2023-04-18 22:56:19,735 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting e4015a6e8a1c46d5b5c385c73017bf2e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1681858574484 2023-04-18 22:56:19,735 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting a47d1ce870ae4d0d9b8103e9138698e9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1681858577788 2023-04-18 22:56:19,735 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting f970ec7d19d44f788760be726752346b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1681858576667 2023-04-18 22:56:19,736 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting c04e572246db4859b25332f9d9ee86d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1681858577788 2023-04-18 22:56:19,736 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 20e31f018d21473fb7ccc72315b0f701, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1681858578839 2023-04-18 22:56:19,736 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 76174215266648d38f78e27802e3b662, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1681858578839 2023-04-18 22:56:19,750 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#B#compaction#566 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:19,752 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#A#compaction#565 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:19,775 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/4f21d747833e423d8389eead4c62f629 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4f21d747833e423d8389eead4c62f629 2023-04-18 22:56:19,779 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8181c42622934944fa97cf06c7004a78/A of 8181c42622934944fa97cf06c7004a78 into 4f21d747833e423d8389eead4c62f629(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:19,779 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:19,779 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/A, priority=11, startTime=1681858579731; duration=0sec 2023-04-18 22:56:19,780 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:19,780 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-18 22:56:19,781 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59202 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-18 22:56:19,781 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/C is initiating minor compaction (all files) 2023-04-18 22:56:19,781 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/C in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:19,782 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/e9756b57bb964725a38da3d5ce2067b5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/dca728a880eb4ba5b3924447c992b3f2, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/1eae5d3a839c45ca88a6cbe3045cba0a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/b1fde4b3f0f545fe920d558efaf0e310, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/25bbed8dd3b44cd989a6ec2175c7ddae] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=57.8 K 2023-04-18 22:56:19,782 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting e9756b57bb964725a38da3d5ce2067b5, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=511, earliestPutTs=1681858574437 2023-04-18 22:56:19,782 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting dca728a880eb4ba5b3924447c992b3f2, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1681858574484 2023-04-18 22:56:19,783 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 1eae5d3a839c45ca88a6cbe3045cba0a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1681858576667 2023-04-18 22:56:19,783 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting b1fde4b3f0f545fe920d558efaf0e310, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=570, earliestPutTs=1681858577788 2023-04-18 22:56:19,784 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 25bbed8dd3b44cd989a6ec2175c7ddae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1681858578839 2023-04-18 22:56:19,793 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#C#compaction#567 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:19,804 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:19,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:19,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:19,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:19,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:19,809 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:19,809 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:19,809 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:19,809 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:19,810 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,810 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,811 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:19,811 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:19,812 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:19,812 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:19,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:19,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:19,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:19,813 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:19,813 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:19,813 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:19,813 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:19,813 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:19,813 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:19,813 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:19,813 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-18 22:56:19,814 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:19,814 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:19,814 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:19,814 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:19,814 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:19,814 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:20,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:20,033 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:20,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:20,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858640046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:20,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858640047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:20,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858640047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:20,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858640048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:20,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858640051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:20,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858640150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:20,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858640150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,162 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/81606f42c2354701b48baa9126eeda7e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/81606f42c2354701b48baa9126eeda7e 2023-04-18 22:56:20,166 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8181c42622934944fa97cf06c7004a78/B of 8181c42622934944fa97cf06c7004a78 into 81606f42c2354701b48baa9126eeda7e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:20,166 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:20,166 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/B, priority=11, startTime=1681858579732; duration=0sec 2023-04-18 22:56:20,166 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:20,214 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/dcbaafbb6e1e4d93b3215b41a32578b3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/dcbaafbb6e1e4d93b3215b41a32578b3 2023-04-18 22:56:20,218 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8181c42622934944fa97cf06c7004a78/C of 8181c42622934944fa97cf06c7004a78 into dcbaafbb6e1e4d93b3215b41a32578b3(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:20,218 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:20,218 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/C, priority=11, startTime=1681858579732; duration=0sec 2023-04-18 22:56:20,218 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:20,237 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304186a3eb57e5cfd4c8db46431aa88ee848a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186a3eb57e5cfd4c8db46431aa88ee848a 2023-04-18 22:56:20,242 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=617, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/71bf2a4eca7f4639be5960bbde046f97 2023-04-18 22:56:20,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:20,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858640355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:20,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858640355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,653 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/d44d1fca90fd4baf8f477e1536b15831 2023-04-18 22:56:20,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:20,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858640658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:20,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858640660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,663 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/97a57833622a40d984e0179ebb8e77b9 2023-04-18 22:56:20,667 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/71bf2a4eca7f4639be5960bbde046f97 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/71bf2a4eca7f4639be5960bbde046f97 2023-04-18 22:56:20,670 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/71bf2a4eca7f4639be5960bbde046f97, entries=150, sequenceid=617, filesize=25.4 K 2023-04-18 22:56:20,670 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/d44d1fca90fd4baf8f477e1536b15831 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/d44d1fca90fd4baf8f477e1536b15831 2023-04-18 22:56:20,674 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/d44d1fca90fd4baf8f477e1536b15831, entries=150, sequenceid=617, filesize=11.9 K 2023-04-18 22:56:20,675 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/97a57833622a40d984e0179ebb8e77b9 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/97a57833622a40d984e0179ebb8e77b9 2023-04-18 22:56:20,678 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/97a57833622a40d984e0179ebb8e77b9, entries=150, sequenceid=617, filesize=11.9 K 2023-04-18 22:56:20,679 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 8181c42622934944fa97cf06c7004a78 in 866ms, sequenceid=617, compaction requested=false 2023-04-18 22:56:20,679 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:20,679 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:20,679 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:20,679 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:20,679 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,679 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:20,679 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:20,681 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,681 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:20,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:20,681 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:20,681 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:20,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:20,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:20,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@579ab28d[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:20,683 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:20,683 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:20,685 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:20,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:20,685 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,685 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:20,686 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:20,686 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:20,686 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:20,686 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:20,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:20,689 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:20,689 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:20,689 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:20,689 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:20,689 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:20,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:20,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:20,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:20,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:20,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:20,692 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:20,694 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:20,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:20,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:20,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:20,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:20,696 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:20,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:20,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:20,696 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:20,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:20,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:20,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:20,696 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:20,697 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:20,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:20,797 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:20,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:20,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:20,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:20,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:20,801 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:20,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:20,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:20,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:20,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:20,801 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:20,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,803 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:20,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:20,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:20,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:20,803 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:20,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:20,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:20,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:20,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:20,805 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:20,805 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:20,805 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:20,808 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:20,808 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:20,808 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:20,808 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:20,808 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:56:20,808 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:20,808 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:20,808 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:20,808 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:20,808 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:20,809 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:21,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:21,055 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:21,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858581070, max timestamp=1681858581076Num uniques 859; 2023-04-18 22:56:21,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:56:21,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858581070, max timestamp=1681858581076Num uniques 859; 2023-04-18 22:56:21,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:56:21,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858581070, max timestamp=1681858581076Num uniques 859; 2023-04-18 22:56:21,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:56:21,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858641096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858641097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858641098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858641163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858641166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858641201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858641201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858641201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,233 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418644a5d04ce9840d2ab25f6df441d04fa to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418644a5d04ce9840d2ab25f6df441d04fa 2023-04-18 22:56:21,239 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=629, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/6e2963c286a34dd39f30ec07de40831d 2023-04-18 22:56:21,249 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=629 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/76461e26f34d437e9e3ccb0b3f07f1c8 2023-04-18 22:56:21,260 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=629 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/5e7cd4da8c7245bbbbc3de4e99076c12 2023-04-18 22:56:21,264 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/6e2963c286a34dd39f30ec07de40831d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/6e2963c286a34dd39f30ec07de40831d 2023-04-18 22:56:21,268 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/6e2963c286a34dd39f30ec07de40831d, entries=100, sequenceid=629, filesize=18.6 K 2023-04-18 22:56:21,269 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/76461e26f34d437e9e3ccb0b3f07f1c8 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/76461e26f34d437e9e3ccb0b3f07f1c8 2023-04-18 22:56:21,272 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/76461e26f34d437e9e3ccb0b3f07f1c8, entries=100, sequenceid=629, filesize=9.5 K 2023-04-18 22:56:21,272 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/5e7cd4da8c7245bbbbc3de4e99076c12 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/5e7cd4da8c7245bbbbc3de4e99076c12 2023-04-18 22:56:21,275 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/5e7cd4da8c7245bbbbc3de4e99076c12, entries=100, sequenceid=629, filesize=9.5 K 2023-04-18 22:56:21,276 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 8181c42622934944fa97cf06c7004a78 in 468ms, sequenceid=629, compaction requested=true 2023-04-18 22:56:21,276 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:21,276 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:21,278 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,278 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,279 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,279 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:21,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bf50d1c[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:21,281 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:21,281 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,285 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:21,286 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,286 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:21,287 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,287 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,287 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:21,287 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:21,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:21,290 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:21,290 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:21,290 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:21,290 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:21,290 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:21,293 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,293 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,293 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:21,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:21,294 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,295 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,298 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:21,298 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:21,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:21,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:21,298 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:21,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:21,298 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:21,298 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,299 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,299 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,299 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,299 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,299 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:21,299 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:21,299 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:21,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:21,399 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:21,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:21,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:21,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:21,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:21,403 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:21,403 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:21,403 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:21,403 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:21,403 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,403 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,405 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:21,405 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:21,407 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-18 22:56:21,407 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:21,407 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:56:21,407 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:21,407 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:56:21,407 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:21,407 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-18 22:56:21,409 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,409 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:21,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:21,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:21,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:21,410 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:21,410 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:21,410 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:21,410 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:21,410 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:21,410 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:21,410 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:21,411 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:21,411 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:21,413 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,413 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,413 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,414 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:21,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7c2364[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:21,415 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:21,415 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,417 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:21,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:21,417 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,417 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:21,417 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:21,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,419 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858641414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,419 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:21,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858641413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,420 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:21,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858641418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,420 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:21,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,421 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304180c4f2cecd9c84342a78c52f1e21818ef to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304180c4f2cecd9c84342a78c52f1e21818ef 2023-04-18 22:56:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,426 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=668, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/e3363ea11ee348bd91f18599390e919f 2023-04-18 22:56:21,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:21,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:21,426 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:21,426 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:21,426 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:21,426 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:21,426 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:21,429 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,429 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,429 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:21,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:21,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:21,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,430 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,432 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,438 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,438 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,438 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,438 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,438 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,438 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=668 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/859c5501c9114faab71cef25cab15d80 2023-04-18 22:56:21,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:21,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:21,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:21,438 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:21,438 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:21,438 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:21,438 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,438 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:21,438 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:21,439 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:21,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:21,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:21,448 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=668 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/4051a18a53ce4581bf11030e33ed9c6f 2023-04-18 22:56:21,452 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/e3363ea11ee348bd91f18599390e919f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e3363ea11ee348bd91f18599390e919f 2023-04-18 22:56:21,455 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e3363ea11ee348bd91f18599390e919f, entries=150, sequenceid=668, filesize=25.4 K 2023-04-18 22:56:21,456 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/859c5501c9114faab71cef25cab15d80 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/859c5501c9114faab71cef25cab15d80 2023-04-18 22:56:21,460 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/859c5501c9114faab71cef25cab15d80, entries=150, sequenceid=668, filesize=11.9 K 2023-04-18 22:56:21,460 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/4051a18a53ce4581bf11030e33ed9c6f as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/4051a18a53ce4581bf11030e33ed9c6f 2023-04-18 22:56:21,465 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/4051a18a53ce4581bf11030e33ed9c6f, entries=150, sequenceid=668, filesize=11.9 K 2023-04-18 22:56:21,466 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~425.48 KB/435696, currentSize=33.54 KB/34350 for 8181c42622934944fa97cf06c7004a78 in 59ms, sequenceid=668, compaction requested=true 2023-04-18 22:56:21,466 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:21,466 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:21,467 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:21,467 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:56:21,467 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:56:21,467 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:21,468 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47186 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:56:21,468 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/B is initiating minor compaction (all files) 2023-04-18 22:56:21,468 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/B in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:21,468 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/81606f42c2354701b48baa9126eeda7e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/d44d1fca90fd4baf8f477e1536b15831, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/76461e26f34d437e9e3ccb0b3f07f1c8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/859c5501c9114faab71cef25cab15d80] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=46.1 K 2023-04-18 22:56:21,468 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 98022 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:56:21,468 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/A is initiating minor compaction (all files) 2023-04-18 22:56:21,468 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/A in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:21,468 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 81606f42c2354701b48baa9126eeda7e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1681858578839 2023-04-18 22:56:21,468 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4f21d747833e423d8389eead4c62f629, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/71bf2a4eca7f4639be5960bbde046f97, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/6e2963c286a34dd39f30ec07de40831d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e3363ea11ee348bd91f18599390e919f] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=95.7 K 2023-04-18 22:56:21,469 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting d44d1fca90fd4baf8f477e1536b15831, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1681858578921 2023-04-18 22:56:21,469 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 4f21d747833e423d8389eead4c62f629, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1681858578839 2023-04-18 22:56:21,469 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 76461e26f34d437e9e3ccb0b3f07f1c8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=629, earliestPutTs=1681858580045 2023-04-18 22:56:21,469 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 71bf2a4eca7f4639be5960bbde046f97, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1681858578921 2023-04-18 22:56:21,470 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 859c5501c9114faab71cef25cab15d80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=668, earliestPutTs=1681858581096 2023-04-18 22:56:21,470 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting 6e2963c286a34dd39f30ec07de40831d, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=629, earliestPutTs=1681858580045 2023-04-18 22:56:21,470 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] compactions.Compactor(207): Compacting e3363ea11ee348bd91f18599390e919f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=668, earliestPutTs=1681858581096 2023-04-18 22:56:21,481 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#B#compaction#578 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:21,482 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#A#compaction#577 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:21,494 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/10f31c27294e4561a71b9285a2321a40 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/10f31c27294e4561a71b9285a2321a40 2023-04-18 22:56:21,499 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8181c42622934944fa97cf06c7004a78/B of 8181c42622934944fa97cf06c7004a78 into 10f31c27294e4561a71b9285a2321a40(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:21,499 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:21,499 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/B, priority=12, startTime=1681858581466; duration=0sec 2023-04-18 22:56:21,499 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-18 22:56:21,499 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-18 22:56:21,501 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47186 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-18 22:56:21,501 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1912): 8181c42622934944fa97cf06c7004a78/C is initiating minor compaction (all files) 2023-04-18 22:56:21,501 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8181c42622934944fa97cf06c7004a78/C in TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:21,501 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/dcbaafbb6e1e4d93b3215b41a32578b3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/97a57833622a40d984e0179ebb8e77b9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/5e7cd4da8c7245bbbbc3de4e99076c12, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/4051a18a53ce4581bf11030e33ed9c6f] into tmpdir=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp, totalSize=46.1 K 2023-04-18 22:56:21,501 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting dcbaafbb6e1e4d93b3215b41a32578b3, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1681858578839 2023-04-18 22:56:21,502 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 97a57833622a40d984e0179ebb8e77b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1681858578921 2023-04-18 22:56:21,502 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 5e7cd4da8c7245bbbbc3de4e99076c12, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=629, earliestPutTs=1681858580045 2023-04-18 22:56:21,503 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] compactions.Compactor(207): Compacting 4051a18a53ce4581bf11030e33ed9c6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=668, earliestPutTs=1681858581096 2023-04-18 22:56:21,510 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] throttle.PressureAwareThroughputController(145): 8181c42622934944fa97cf06c7004a78#C#compaction#579 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-18 22:56:21,524 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/26daface7a404826b2efdd06bb85aef3 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/26daface7a404826b2efdd06bb85aef3 2023-04-18 22:56:21,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:21,527 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-18 22:56:21,528 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:21,528 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:21,528 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:21,528 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:21,528 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:21,528 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:21,529 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8181c42622934944fa97cf06c7004a78/C of 8181c42622934944fa97cf06c7004a78 into 26daface7a404826b2efdd06bb85aef3(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:21,529 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:21,529 INFO [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/C, priority=12, startTime=1681858581467; duration=0sec 2023-04-18 22:56:21,529 DEBUG [RS:0;jenkins-hbase4:35971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:21,539 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:21,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:21,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:21,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:21,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:21,544 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:21,544 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:21,544 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:21,544 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:21,544 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,545 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,549 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,549 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:21,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:21,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:21,549 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,549 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:21,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:21,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:21,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:21,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:21,550 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:21,550 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:21,550 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:21,550 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:21,550 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:21,550 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:21,550 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:21,550 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:21,551 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:21,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:21,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:21,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:21,551 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:21,553 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,554 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:21,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:21,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28fc7cfc[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:21,556 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:21,556 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,557 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041827e103b9221b42828ca636d22cf38a7b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041827e103b9221b42828ca636d22cf38a7b 2023-04-18 22:56:21,558 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858641556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858581551, max timestamp=1681858581557Num uniques 859; 2023-04-18 22:56:21,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:56:21,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681858581551, max timestamp=1681858581557Num uniques 859; 2023-04-18 22:56:21,559 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:21,559 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:21,559 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:21,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-18 22:56:21,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681858581551, max timestamp=1681858581557Num uniques 859; 2023-04-18 22:56:21,559 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,560 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:21,561 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:21,561 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:21,561 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:21,561 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-18 22:56:21,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:21,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:21,564 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:21,564 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:21,564 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:21,564 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:21,564 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:21,566 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=680, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/104b8918dff94c3093e8576d0c896b7e 2023-04-18 22:56:21,567 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,567 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,567 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:21,568 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,571 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:21,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:21,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:21,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:21,575 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,576 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:21,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:21,576 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:21,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:21,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:21,577 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:21,577 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:21,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=680 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/7797ef858a59487aab106868437bea03 2023-04-18 22:56:21,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858641586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858641587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858641660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,677 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:21,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:21,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:21,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:21,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:21,680 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:21,680 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:21,680 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:21,680 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:21,680 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,680 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,682 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,682 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:21,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:21,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:21,682 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,682 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:21,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:21,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:21,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:21,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:21,683 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:21,683 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:21,683 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:21,683 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:21,683 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:21,683 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:21,684 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:21,684 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:21,684 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:21,684 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:21,684 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:21,684 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,684 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:21,684 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:21,685 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,685 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,686 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,686 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:21,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:21,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a70bdb4[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:21,687 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:21,687 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,688 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,689 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:21,689 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:21,689 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:21,689 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,689 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:21,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858641689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,692 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:21,692 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:21,692 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:21,692 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858641692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:21,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:21,695 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:21,695 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:21,695 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:21,695 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:21,695 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:21,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:21,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,697 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,698 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,699 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,701 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,701 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,701 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,702 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:21,701 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:21,702 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:21,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:21,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:21,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:21,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:21,702 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:21,702 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:21,702 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,702 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,702 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,702 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,802 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:21,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:21,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:21,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:21,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:21,805 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:21,805 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:21,806 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:21,806 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:21,806 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,806 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,807 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:21,807 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:21,808 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,808 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:21,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:21,809 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:21,809 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:21,809 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:21,810 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:21,810 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:21,810 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:21,810 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:21,810 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:21,810 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:21,810 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:21,810 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:21,810 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,810 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:21,810 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:21,813 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,813 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,813 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,813 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19dec80b[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:21,814 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:21,815 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,815 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,816 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:21,816 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:21,816 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,816 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:21,816 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:21,817 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:21,817 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,817 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,817 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:21,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:21,820 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:21,820 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:21,820 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:21,820 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:21,820 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:21,822 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,822 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,822 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:21,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,823 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,824 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,827 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:21,827 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:21,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:21,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:21,827 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:21,828 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:21,828 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:21,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:21,828 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,828 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,828 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,828 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,828 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:21,828 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,828 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:21,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858641864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858641892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:21,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858641896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,904 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/0841f53a2baa4b33b6bd9212a7774c94 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/0841f53a2baa4b33b6bd9212a7774c94 2023-04-18 22:56:21,908 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8181c42622934944fa97cf06c7004a78/A of 8181c42622934944fa97cf06c7004a78 into 0841f53a2baa4b33b6bd9212a7774c94(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-18 22:56:21,908 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:21,909 INFO [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78., storeName=8181c42622934944fa97cf06c7004a78/A, priority=12, startTime=1681858581466; duration=0sec 2023-04-18 22:56:21,909 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:21,928 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:21,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:21,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:21,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:21,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:21,931 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:21,931 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:21,931 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:21,931 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:21,931 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,932 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,933 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:21,933 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:21,933 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,933 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:21,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:21,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:21,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:21,937 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:21,937 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:21,937 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:21,938 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:21,938 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:21,938 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:21,938 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:21,940 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,940 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,940 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,940 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:21,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4767af57[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:21,941 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:21,942 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,943 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,943 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:21,943 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:21,943 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:21,943 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,943 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:21,945 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,945 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:21,945 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:21,945 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:21,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:21,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:21,947 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:21,947 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:21,947 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:21,947 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:21,948 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:21,949 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:21,949 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,949 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:21,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:21,949 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:21,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:21,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:21,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:21,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,951 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:21,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:21,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,953 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:21,953 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:21,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:21,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:21,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:21,953 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:21,953 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:21,954 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,954 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:21,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:21,954 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:21,954 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:21,954 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:21,954 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,954 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:21,954 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:22,002 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=680 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/a340438a2006409eb0c56d5afd337f13 2023-04-18 22:56:22,006 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/104b8918dff94c3093e8576d0c896b7e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/104b8918dff94c3093e8576d0c896b7e 2023-04-18 22:56:22,009 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/104b8918dff94c3093e8576d0c896b7e, entries=250, sequenceid=680, filesize=39.2 K 2023-04-18 22:56:22,010 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/7797ef858a59487aab106868437bea03 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/7797ef858a59487aab106868437bea03 2023-04-18 22:56:22,013 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/7797ef858a59487aab106868437bea03, entries=150, sequenceid=680, filesize=11.9 K 2023-04-18 22:56:22,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/a340438a2006409eb0c56d5afd337f13 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/a340438a2006409eb0c56d5afd337f13 2023-04-18 22:56:22,017 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/a340438a2006409eb0c56d5afd337f13, entries=150, sequenceid=680, filesize=11.9 K 2023-04-18 22:56:22,018 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 8181c42622934944fa97cf06c7004a78 in 491ms, sequenceid=680, compaction requested=false 2023-04-18 22:56:22,018 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:22,054 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:22,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:22,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-18 22:56:22,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-18 22:56:22,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-18 22:56:22,060 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-18 22:56:22,060 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:22,060 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-18 22:56:22,060 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-18 22:56:22,061 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:22,061 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:22,063 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,063 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:22,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:22,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:22,063 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,063 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-18 22:56:22,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:22,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-18 22:56:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:22,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-18 22:56:22,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-18 22:56:22,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-18 22:56:22,064 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-18 22:56:22,064 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-18 22:56:22,064 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-18 22:56:22,064 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-18 22:56:22,065 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-18 22:56:22,065 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:22,065 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. started... 2023-04-18 22:56:22,065 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=125.23 KB heapSize=355.30 KB 2023-04-18 22:56:22,065 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:22,065 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:22,065 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:22,065 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:22,065 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:22,065 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-18 22:56:22,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:22,170 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. as already flushing 2023-04-18 22:56:22,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:22,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858642179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:22,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:22,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858642180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858642180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:22,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858642197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:22,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858642201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:22,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858642283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:22,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858642285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:22,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858642285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,368 DEBUG [Thread-3647] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c53bf96 to 127.0.0.1:50825 2023-04-18 22:56:22,368 DEBUG [Thread-3651] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d765193 to 127.0.0.1:50825 2023-04-18 22:56:22,369 DEBUG [Thread-3647] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:56:22,369 DEBUG [Thread-3651] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:56:22,370 DEBUG [Thread-3649] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b59bdcf to 127.0.0.1:50825 2023-04-18 22:56:22,370 DEBUG [Thread-3649] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:56:22,371 DEBUG [Thread-3655] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28164190 to 127.0.0.1:50825 2023-04-18 22:56:22,371 DEBUG [Thread-3655] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:56:22,371 DEBUG [Thread-3653] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32bf16cd to 127.0.0.1:50825 2023-04-18 22:56:22,371 DEBUG [Thread-3653] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:56:22,481 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230418dafefc8be5014aa4b429b068c741769c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418dafefc8be5014aa4b429b068c741769c 2023-04-18 22:56:22,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:22,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858642488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:22,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858642489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:22,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858642489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:22,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858642701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:22,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858642704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:22,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:22,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35971] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858642791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858642791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:22,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858642791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:22,885 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=717, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/4596264e40fb4f3c84ec9c058f84c47e 2023-04-18 22:56:23,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:23,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45860 deadline: 1681858643292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,295 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=717 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/bc5db5f4e49145f599b2755626b949ce 2023-04-18 22:56:23,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:23,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35971] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45852 deadline: 1681858643296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:23,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45876 deadline: 1681858643296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,703 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=717 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/e5a71da7015241daaa11c03dc39b557d 2023-04-18 22:56:23,706 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/4596264e40fb4f3c84ec9c058f84c47e as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4596264e40fb4f3c84ec9c058f84c47e 2023-04-18 22:56:23,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:23,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45800 deadline: 1681858643709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,710 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4596264e40fb4f3c84ec9c058f84c47e, entries=150, sequenceid=717, filesize=25.4 K 2023-04-18 22:56:23,711 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/bc5db5f4e49145f599b2755626b949ce as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/bc5db5f4e49145f599b2755626b949ce 2023-04-18 22:56:23,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-18 22:56:23,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:45846 deadline: 1681858643712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,714 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/bc5db5f4e49145f599b2755626b949ce, entries=150, sequenceid=717, filesize=11.9 K 2023-04-18 22:56:23,715 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/e5a71da7015241daaa11c03dc39b557d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/e5a71da7015241daaa11c03dc39b557d 2023-04-18 22:56:23,718 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/e5a71da7015241daaa11c03dc39b557d, entries=150, sequenceid=717, filesize=11.9 K 2023-04-18 22:56:23,718 INFO [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~355.25 KB/363776, currentSize=53.67 KB/54960 for 8181c42622934944fa97cf06c7004a78 in 1653ms, sequenceid=717, compaction requested=true 2023-04-18 22:56:23,718 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:23,718 DEBUG [rs(jenkins-hbase4.apache.org,35971,1681858424869)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:23,718 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-18 22:56:23,719 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-18 22:56:23,719 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,719 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-18 22:56:23,719 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-18 22:56:23,723 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,723 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:23,723 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:23,723 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-18 22:56:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:23,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35971,1681858424869' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-18 22:56:23,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c3730db[Count = 0] remaining members to acquire global barrier 2023-04-18 22:56:23,725 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-18 22:56:23,725 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:23,726 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:23,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:23,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:23,726 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-18 22:56:23,726 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-18 22:56:23,726 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,726 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-18 22:56:23,726 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35971,1681858424869' in zk 2023-04-18 22:56:23,727 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-18 22:56:23,727 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,727 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:23,728 DEBUG [member: 'jenkins-hbase4.apache.org,35971,1681858424869' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-18 22:56:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35971,1681858424869': 2023-04-18 22:56:23,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35971,1681858424869' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-18 22:56:23,730 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-18 22:56:23,730 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-18 22:56:23,730 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-18 22:56:23,730 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-18 22:56:23,730 INFO [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-18 22:56:23,732 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:23,732 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-18 22:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-18 22:56:23,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:23,732 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-18 22:56:23,732 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-18 22:56:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:23,733 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-18 22:56:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-18 22:56:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,736 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,736 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-18 22:56:23,736 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:23,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-18 22:56:23,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-18 22:56:23,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-18 22:56:23,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-18 22:56:23,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-18 22:56:23,736 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-18 22:56:23,736 DEBUG [(jenkins-hbase4.apache.org,46739,1681858423665)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-18 22:56:23,736 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-18 22:56:23,736 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:23,737 DEBUG [Thread-3646] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-18 22:56:23,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-18 22:56:23,737 DEBUG [Thread-3646] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-18 22:56:23,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-18 22:56:23,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:23,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-18 22:56:23,737 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-18 22:56:23,837 DEBUG [Thread-3646] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-18 22:56:23,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-18 22:56:24,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35971] regionserver.HRegion(9158): Flush requested on 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:24,298 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-18 22:56:24,298 DEBUG [Thread-3640] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4db22237 to 127.0.0.1:50825 2023-04-18 22:56:24,298 DEBUG [Thread-3642] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08b57aae to 127.0.0.1:50825 2023-04-18 22:56:24,298 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:24,298 DEBUG [Thread-3642] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:56:24,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:24,298 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:24,298 DEBUG [Thread-3640] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:56:24,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:24,299 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:24,299 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:24,300 DEBUG [Thread-3638] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e8654 to 127.0.0.1:50825 2023-04-18 22:56:24,300 DEBUG [Thread-3638] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:56:24,709 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304183c8dcb315caf42bdbe3b4d9c3ceb059e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304183c8dcb315caf42bdbe3b4d9c3ceb059e 2023-04-18 22:56:25,114 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=730, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/f669cfd8747242b39ec927ad03309d6d 2023-04-18 22:56:25,523 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=730 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/c8c8d9b498a7470ea7e6b55295f2051c 2023-04-18 22:56:25,720 DEBUG [Thread-3636] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f06dd50 to 127.0.0.1:50825 2023-04-18 22:56:25,720 DEBUG [Thread-3636] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:56:25,730 DEBUG [Thread-3644] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06799455 to 127.0.0.1:50825 2023-04-18 22:56:25,730 DEBUG [Thread-3644] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:56:25,730 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-18 22:56:25,730 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 93 2023-04-18 22:56:25,730 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 70 2023-04-18 22:56:25,730 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 196 2023-04-18 22:56:25,730 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 169 2023-04-18 22:56:25,730 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(394): wrote 90 2023-04-18 22:56:25,730 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-18 22:56:25,730 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(398): read 4195 2023-04-18 22:56:25,730 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(398): read 4276 2023-04-18 22:56:25,731 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(398): read 4294 2023-04-18 22:56:25,731 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(398): read 4192 2023-04-18 22:56:25,731 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(398): read 4271 2023-04-18 22:56:25,731 INFO [Listener at localhost/36651] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-18 22:56:25,731 INFO [Listener at localhost/36651] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-18 22:56:25,731 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e3503b4 to 127.0.0.1:50825 2023-04-18 22:56:25,731 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:56:25,731 INFO [Listener at localhost/36651] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-18 22:56:25,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-18 22:56:25,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-18 22:56:25,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-18 22:56:25,735 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858585735"}]},"ts":"1681858585735"} 2023-04-18 22:56:25,736 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-18 22:56:25,738 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-18 22:56:25,738 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8181c42622934944fa97cf06c7004a78, UNASSIGN}] 2023-04-18 22:56:25,740 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8181c42622934944fa97cf06c7004a78, UNASSIGN 2023-04-18 22:56:25,740 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=8181c42622934944fa97cf06c7004a78, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:25,740 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858585740"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681858585740"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681858585740"}]},"ts":"1681858585740"} 2023-04-18 22:56:25,741 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869}] 2023-04-18 22:56:25,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-18 22:56:25,893 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:25,893 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8181c42622934944fa97cf06c7004a78, disabling compactions & flushes 2023-04-18 22:56:25,893 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:25,932 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=730 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/233d045851904ab0a366d262ab6f0fd8 2023-04-18 22:56:25,935 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/f669cfd8747242b39ec927ad03309d6d as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/f669cfd8747242b39ec927ad03309d6d 2023-04-18 22:56:25,937 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/f669cfd8747242b39ec927ad03309d6d, entries=150, sequenceid=730, filesize=25.4 K 2023-04-18 22:56:25,938 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/c8c8d9b498a7470ea7e6b55295f2051c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/c8c8d9b498a7470ea7e6b55295f2051c 2023-04-18 22:56:25,941 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/c8c8d9b498a7470ea7e6b55295f2051c, entries=150, sequenceid=730, filesize=11.9 K 2023-04-18 22:56:25,941 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/233d045851904ab0a366d262ab6f0fd8 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/233d045851904ab0a366d262ab6f0fd8 2023-04-18 22:56:25,944 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/233d045851904ab0a366d262ab6f0fd8, entries=150, sequenceid=730, filesize=11.9 K 2023-04-18 22:56:25,945 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=20.13 KB/20610 for 8181c42622934944fa97cf06c7004a78 in 1647ms, sequenceid=730, compaction requested=true 2023-04-18 22:56:25,945 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:25,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:25,945 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. because compaction request was cancelled 2023-04-18 22:56:25,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:25,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. after waiting 0 ms 2023-04-18 22:56:25,945 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:25,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:25,945 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. because compaction request was cancelled 2023-04-18 22:56:25,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 8181c42622934944fa97cf06c7004a78 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-18 22:56:25,945 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:25,945 DEBUG [RS:0;jenkins-hbase4:35971-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. because compaction request was cancelled 2023-04-18 22:56:25,945 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-18 22:56:25,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=A 2023-04-18 22:56:25,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:25,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=B 2023-04-18 22:56:25,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:25,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8181c42622934944fa97cf06c7004a78, store=C 2023-04-18 22:56:25,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-18 22:56:26,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-18 22:56:26,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-18 22:56:26,355 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304182014e477149c4fe198490f2b6eaef925 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304182014e477149c4fe198490f2b6eaef925 2023-04-18 22:56:26,760 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=736, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/00939b41c5cd4892b47110e791fd92b6 2023-04-18 22:56:26,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-18 22:56:27,168 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=736 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/afa38f257e0844afa03410a93f276dc7 2023-04-18 22:56:27,176 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=736 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/fe8d6bb133694e64ae8d62ccf77f3f8c 2023-04-18 22:56:27,179 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/A/00939b41c5cd4892b47110e791fd92b6 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/00939b41c5cd4892b47110e791fd92b6 2023-04-18 22:56:27,181 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/00939b41c5cd4892b47110e791fd92b6, entries=150, sequenceid=736, filesize=25.4 K 2023-04-18 22:56:27,182 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/B/afa38f257e0844afa03410a93f276dc7 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/afa38f257e0844afa03410a93f276dc7 2023-04-18 22:56:27,185 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/afa38f257e0844afa03410a93f276dc7, entries=150, sequenceid=736, filesize=11.9 K 2023-04-18 22:56:27,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/.tmp/C/fe8d6bb133694e64ae8d62ccf77f3f8c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/fe8d6bb133694e64ae8d62ccf77f3f8c 2023-04-18 22:56:27,188 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/fe8d6bb133694e64ae8d62ccf77f3f8c, entries=150, sequenceid=736, filesize=11.9 K 2023-04-18 22:56:27,189 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 8181c42622934944fa97cf06c7004a78 in 1244ms, sequenceid=736, compaction requested=true 2023-04-18 22:56:27,190 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/610f19eb126d49fcbfe93b92359eaaf9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/baffd83b791a40048c07bf232167243d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e4a809f9509b42a3bfaa677798307ce1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/cbc3c4b69d7e43d7beec3143053a2f3b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/564279ea5ab743579c69e4f7f920680f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/cc878e24d7f24b159221bf03392b1805, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/58bd9d6612574846bdbe896a05598eac, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/d151bf2efe1d48308a7617c53b2a6029, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e81e8f9a5b964cb69ddc47793a626d0b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/61478ab4cc934eaf8d90955856badb23, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/f5ac854c2d4740bea8f8d93af41a6787, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/828ac514d2364cb0ae4a58846167e982, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/d7c662601feb47569d1d124531c4bca0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/c5ade2f2104a41b38430ebe872c3f72f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/fd2aa31b472f4f10b07694edbf132a3f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4ee4ba3abe8b402dbd123d83388d7c60, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/9ecd515dae9e48ca8abdf72718dc331f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/ad9466a783bd4113b5f94da704bb25d5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/b0069692dcdd4c909cbe0704ddddba62, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/8f4e9f9a6c4245869a5e982a5726cd45, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4fd3eb43719941fb950c8671bb06f48d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/d4201c055ec5465a9eeda5bbcca1f555, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/5efc5efe80dd4f228207b7734bf85939, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/3c1de124c9b9479a8f4e11c28ac7bf33, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/a47d1ce870ae4d0d9b8103e9138698e9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4f21d747833e423d8389eead4c62f629, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/20e31f018d21473fb7ccc72315b0f701, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/71bf2a4eca7f4639be5960bbde046f97, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/6e2963c286a34dd39f30ec07de40831d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e3363ea11ee348bd91f18599390e919f] to archive 2023-04-18 22:56:27,191 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:56:27,192 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/610f19eb126d49fcbfe93b92359eaaf9 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/610f19eb126d49fcbfe93b92359eaaf9 2023-04-18 22:56:27,193 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/baffd83b791a40048c07bf232167243d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/baffd83b791a40048c07bf232167243d 2023-04-18 22:56:27,194 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e4a809f9509b42a3bfaa677798307ce1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e4a809f9509b42a3bfaa677798307ce1 2023-04-18 22:56:27,195 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/cbc3c4b69d7e43d7beec3143053a2f3b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/cbc3c4b69d7e43d7beec3143053a2f3b 2023-04-18 22:56:27,196 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/564279ea5ab743579c69e4f7f920680f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/564279ea5ab743579c69e4f7f920680f 2023-04-18 22:56:27,197 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/cc878e24d7f24b159221bf03392b1805 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/cc878e24d7f24b159221bf03392b1805 2023-04-18 22:56:27,198 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/58bd9d6612574846bdbe896a05598eac to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/58bd9d6612574846bdbe896a05598eac 2023-04-18 22:56:27,199 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/d151bf2efe1d48308a7617c53b2a6029 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/d151bf2efe1d48308a7617c53b2a6029 2023-04-18 22:56:27,200 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e81e8f9a5b964cb69ddc47793a626d0b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e81e8f9a5b964cb69ddc47793a626d0b 2023-04-18 22:56:27,201 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/61478ab4cc934eaf8d90955856badb23 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/61478ab4cc934eaf8d90955856badb23 2023-04-18 22:56:27,202 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/f5ac854c2d4740bea8f8d93af41a6787 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/f5ac854c2d4740bea8f8d93af41a6787 2023-04-18 22:56:27,203 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/828ac514d2364cb0ae4a58846167e982 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/828ac514d2364cb0ae4a58846167e982 2023-04-18 22:56:27,204 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/d7c662601feb47569d1d124531c4bca0 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/d7c662601feb47569d1d124531c4bca0 2023-04-18 22:56:27,205 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/c5ade2f2104a41b38430ebe872c3f72f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/c5ade2f2104a41b38430ebe872c3f72f 2023-04-18 22:56:27,206 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/fd2aa31b472f4f10b07694edbf132a3f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/fd2aa31b472f4f10b07694edbf132a3f 2023-04-18 22:56:27,208 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4ee4ba3abe8b402dbd123d83388d7c60 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4ee4ba3abe8b402dbd123d83388d7c60 2023-04-18 22:56:27,209 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/9ecd515dae9e48ca8abdf72718dc331f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/9ecd515dae9e48ca8abdf72718dc331f 2023-04-18 22:56:27,210 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/ad9466a783bd4113b5f94da704bb25d5 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/ad9466a783bd4113b5f94da704bb25d5 2023-04-18 22:56:27,211 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/b0069692dcdd4c909cbe0704ddddba62 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/b0069692dcdd4c909cbe0704ddddba62 2023-04-18 22:56:27,212 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/8f4e9f9a6c4245869a5e982a5726cd45 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/8f4e9f9a6c4245869a5e982a5726cd45 2023-04-18 22:56:27,213 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4fd3eb43719941fb950c8671bb06f48d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4fd3eb43719941fb950c8671bb06f48d 2023-04-18 22:56:27,214 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/d4201c055ec5465a9eeda5bbcca1f555 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/d4201c055ec5465a9eeda5bbcca1f555 2023-04-18 22:56:27,215 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/5efc5efe80dd4f228207b7734bf85939 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/5efc5efe80dd4f228207b7734bf85939 2023-04-18 22:56:27,216 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/3c1de124c9b9479a8f4e11c28ac7bf33 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/3c1de124c9b9479a8f4e11c28ac7bf33 2023-04-18 22:56:27,217 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/a47d1ce870ae4d0d9b8103e9138698e9 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/a47d1ce870ae4d0d9b8103e9138698e9 2023-04-18 22:56:27,218 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4f21d747833e423d8389eead4c62f629 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4f21d747833e423d8389eead4c62f629 2023-04-18 22:56:27,219 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/20e31f018d21473fb7ccc72315b0f701 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/20e31f018d21473fb7ccc72315b0f701 2023-04-18 22:56:27,220 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/71bf2a4eca7f4639be5960bbde046f97 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/71bf2a4eca7f4639be5960bbde046f97 2023-04-18 22:56:27,221 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/6e2963c286a34dd39f30ec07de40831d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/6e2963c286a34dd39f30ec07de40831d 2023-04-18 22:56:27,222 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e3363ea11ee348bd91f18599390e919f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/e3363ea11ee348bd91f18599390e919f 2023-04-18 22:56:27,224 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/db372612858246aaa4e907e6c9179558, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/1fa99d3246484e90ad97e46a02fe5ba1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/070ea5ba645745b4b02cd08c2efba7cd, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/73c14ff73b4d4855be9efc912cdf8eb7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/4caccb48ebc340caa21b3d8f946ca0b0, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/5364433954ce4561814e30819c239fb8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/73955fc37e194e778ef596234acc9a11, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/a618d41a785949e2af596c24ec9dd4cf, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f58a5ccc4c1543899ca0acf154b7a9b1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/040adf32319b42f8a6c550d1015aaa3a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/98045c6babbe401b86fc3ca2a22a9ba4, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f0c7e83aa4fd4469b44fcb59de72176c, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/345b5b72331645ba9e642879161de145, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/c1203a835680480ebc4d432c3ba5900a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f581499d082d4841bf4fefc4da276d0e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/42bc8ad60e84413e9138abe4e1d737dc, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/e5db8fe6d5f74b73878b54af650f6257, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/88a612595ce84c21b710bef84803aea2, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/5936017b764846d5adb005e49f7354fe, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/25f04553152d44a59dddc007cf9ac801, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/42d62044040149f3a86b548d01e2226f, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/d497ce3e340140e88b0a1f3871c3e569, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/e4015a6e8a1c46d5b5c385c73017bf2e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f970ec7d19d44f788760be726752346b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/c04e572246db4859b25332f9d9ee86d4, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/81606f42c2354701b48baa9126eeda7e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/76174215266648d38f78e27802e3b662, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/d44d1fca90fd4baf8f477e1536b15831, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/76461e26f34d437e9e3ccb0b3f07f1c8, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/859c5501c9114faab71cef25cab15d80] to archive 2023-04-18 22:56:27,225 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:56:27,226 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/db372612858246aaa4e907e6c9179558 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/db372612858246aaa4e907e6c9179558 2023-04-18 22:56:27,227 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/1fa99d3246484e90ad97e46a02fe5ba1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/1fa99d3246484e90ad97e46a02fe5ba1 2023-04-18 22:56:27,228 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/070ea5ba645745b4b02cd08c2efba7cd to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/070ea5ba645745b4b02cd08c2efba7cd 2023-04-18 22:56:27,229 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/73c14ff73b4d4855be9efc912cdf8eb7 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/73c14ff73b4d4855be9efc912cdf8eb7 2023-04-18 22:56:27,230 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/4caccb48ebc340caa21b3d8f946ca0b0 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/4caccb48ebc340caa21b3d8f946ca0b0 2023-04-18 22:56:27,231 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/5364433954ce4561814e30819c239fb8 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/5364433954ce4561814e30819c239fb8 2023-04-18 22:56:27,231 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/73955fc37e194e778ef596234acc9a11 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/73955fc37e194e778ef596234acc9a11 2023-04-18 22:56:27,232 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/a618d41a785949e2af596c24ec9dd4cf to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/a618d41a785949e2af596c24ec9dd4cf 2023-04-18 22:56:27,233 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f58a5ccc4c1543899ca0acf154b7a9b1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f58a5ccc4c1543899ca0acf154b7a9b1 2023-04-18 22:56:27,234 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/040adf32319b42f8a6c550d1015aaa3a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/040adf32319b42f8a6c550d1015aaa3a 2023-04-18 22:56:27,235 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/98045c6babbe401b86fc3ca2a22a9ba4 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/98045c6babbe401b86fc3ca2a22a9ba4 2023-04-18 22:56:27,236 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f0c7e83aa4fd4469b44fcb59de72176c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f0c7e83aa4fd4469b44fcb59de72176c 2023-04-18 22:56:27,237 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/345b5b72331645ba9e642879161de145 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/345b5b72331645ba9e642879161de145 2023-04-18 22:56:27,238 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/c1203a835680480ebc4d432c3ba5900a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/c1203a835680480ebc4d432c3ba5900a 2023-04-18 22:56:27,239 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f581499d082d4841bf4fefc4da276d0e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f581499d082d4841bf4fefc4da276d0e 2023-04-18 22:56:27,240 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/42bc8ad60e84413e9138abe4e1d737dc to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/42bc8ad60e84413e9138abe4e1d737dc 2023-04-18 22:56:27,241 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/e5db8fe6d5f74b73878b54af650f6257 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/e5db8fe6d5f74b73878b54af650f6257 2023-04-18 22:56:27,242 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/88a612595ce84c21b710bef84803aea2 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/88a612595ce84c21b710bef84803aea2 2023-04-18 22:56:27,243 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/5936017b764846d5adb005e49f7354fe to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/5936017b764846d5adb005e49f7354fe 2023-04-18 22:56:27,243 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/25f04553152d44a59dddc007cf9ac801 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/25f04553152d44a59dddc007cf9ac801 2023-04-18 22:56:27,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/42d62044040149f3a86b548d01e2226f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/42d62044040149f3a86b548d01e2226f 2023-04-18 22:56:27,245 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/d497ce3e340140e88b0a1f3871c3e569 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/d497ce3e340140e88b0a1f3871c3e569 2023-04-18 22:56:27,246 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/e4015a6e8a1c46d5b5c385c73017bf2e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/e4015a6e8a1c46d5b5c385c73017bf2e 2023-04-18 22:56:27,247 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f970ec7d19d44f788760be726752346b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/f970ec7d19d44f788760be726752346b 2023-04-18 22:56:27,248 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/c04e572246db4859b25332f9d9ee86d4 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/c04e572246db4859b25332f9d9ee86d4 2023-04-18 22:56:27,249 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/81606f42c2354701b48baa9126eeda7e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/81606f42c2354701b48baa9126eeda7e 2023-04-18 22:56:27,250 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/76174215266648d38f78e27802e3b662 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/76174215266648d38f78e27802e3b662 2023-04-18 22:56:27,250 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/d44d1fca90fd4baf8f477e1536b15831 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/d44d1fca90fd4baf8f477e1536b15831 2023-04-18 22:56:27,251 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/76461e26f34d437e9e3ccb0b3f07f1c8 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/76461e26f34d437e9e3ccb0b3f07f1c8 2023-04-18 22:56:27,252 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/859c5501c9114faab71cef25cab15d80 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/859c5501c9114faab71cef25cab15d80 2023-04-18 22:56:27,255 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/81fcb5c4ae9442dfb00c2c696f9fcf43, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/1c6a68bd4348419d87ece43fec1e4ee6, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/54826ebf3eae4c38b6e32fb81866e3ef, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/56397fbac6c2440caf04ba166764019d, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/106c7305baeb440283898321451e943a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/3bc6d813bc334a5f8e3215b43aafebd3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/3ef7f5086e1c4ec89c632e7e4513fd28, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/f338123123cb4267a8b620bb28b1e856, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/261d649e79594bf8b89c0071a05fb974, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/589d6f0c2cbf471bb3c020da5375017e, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/e28f02b6eb3b4d6bae1d23b65c5562f7, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/2cc42a5d68ca4e15a95b05eaf09939c1, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/5128c555dc1d40c2b2137acd8184d9ea, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/9f0bf10a3ea44b8c8500d1826f2ce748, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/49d5072797b84e61b9178384f0e33263, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/d7fc7719ec1c4d9c9ad140c97dad48ae, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/43e7a9ffc77641dab32944f4f2a2b324, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/a73bf9fe2ea34f4d86c40f120399f70b, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/b8bfd6f8c0834b51b734da8407f552d5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/d926a0d0d06e499a9c479e92c348b1ea, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/e9756b57bb964725a38da3d5ce2067b5, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/048a250d600445abb38f5c6a452cd329, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/dca728a880eb4ba5b3924447c992b3f2, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/1eae5d3a839c45ca88a6cbe3045cba0a, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/b1fde4b3f0f545fe920d558efaf0e310, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/dcbaafbb6e1e4d93b3215b41a32578b3, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/25bbed8dd3b44cd989a6ec2175c7ddae, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/97a57833622a40d984e0179ebb8e77b9, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/5e7cd4da8c7245bbbbc3de4e99076c12, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/4051a18a53ce4581bf11030e33ed9c6f] to archive 2023-04-18 22:56:27,256 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-18 22:56:27,257 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/81fcb5c4ae9442dfb00c2c696f9fcf43 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/81fcb5c4ae9442dfb00c2c696f9fcf43 2023-04-18 22:56:27,258 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/1c6a68bd4348419d87ece43fec1e4ee6 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/1c6a68bd4348419d87ece43fec1e4ee6 2023-04-18 22:56:27,258 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/54826ebf3eae4c38b6e32fb81866e3ef to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/54826ebf3eae4c38b6e32fb81866e3ef 2023-04-18 22:56:27,259 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/56397fbac6c2440caf04ba166764019d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/56397fbac6c2440caf04ba166764019d 2023-04-18 22:56:27,260 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/106c7305baeb440283898321451e943a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/106c7305baeb440283898321451e943a 2023-04-18 22:56:27,261 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/3bc6d813bc334a5f8e3215b43aafebd3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/3bc6d813bc334a5f8e3215b43aafebd3 2023-04-18 22:56:27,262 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/3ef7f5086e1c4ec89c632e7e4513fd28 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/3ef7f5086e1c4ec89c632e7e4513fd28 2023-04-18 22:56:27,263 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/f338123123cb4267a8b620bb28b1e856 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/f338123123cb4267a8b620bb28b1e856 2023-04-18 22:56:27,264 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/261d649e79594bf8b89c0071a05fb974 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/261d649e79594bf8b89c0071a05fb974 2023-04-18 22:56:27,265 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/589d6f0c2cbf471bb3c020da5375017e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/589d6f0c2cbf471bb3c020da5375017e 2023-04-18 22:56:27,266 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/e28f02b6eb3b4d6bae1d23b65c5562f7 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/e28f02b6eb3b4d6bae1d23b65c5562f7 2023-04-18 22:56:27,267 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/2cc42a5d68ca4e15a95b05eaf09939c1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/2cc42a5d68ca4e15a95b05eaf09939c1 2023-04-18 22:56:27,268 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/5128c555dc1d40c2b2137acd8184d9ea to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/5128c555dc1d40c2b2137acd8184d9ea 2023-04-18 22:56:27,269 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/9f0bf10a3ea44b8c8500d1826f2ce748 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/9f0bf10a3ea44b8c8500d1826f2ce748 2023-04-18 22:56:27,270 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/49d5072797b84e61b9178384f0e33263 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/49d5072797b84e61b9178384f0e33263 2023-04-18 22:56:27,270 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/d7fc7719ec1c4d9c9ad140c97dad48ae to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/d7fc7719ec1c4d9c9ad140c97dad48ae 2023-04-18 22:56:27,271 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/43e7a9ffc77641dab32944f4f2a2b324 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/43e7a9ffc77641dab32944f4f2a2b324 2023-04-18 22:56:27,272 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/a73bf9fe2ea34f4d86c40f120399f70b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/a73bf9fe2ea34f4d86c40f120399f70b 2023-04-18 22:56:27,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/b8bfd6f8c0834b51b734da8407f552d5 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/b8bfd6f8c0834b51b734da8407f552d5 2023-04-18 22:56:27,274 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/d926a0d0d06e499a9c479e92c348b1ea to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/d926a0d0d06e499a9c479e92c348b1ea 2023-04-18 22:56:27,275 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/e9756b57bb964725a38da3d5ce2067b5 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/e9756b57bb964725a38da3d5ce2067b5 2023-04-18 22:56:27,276 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/048a250d600445abb38f5c6a452cd329 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/048a250d600445abb38f5c6a452cd329 2023-04-18 22:56:27,277 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/dca728a880eb4ba5b3924447c992b3f2 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/dca728a880eb4ba5b3924447c992b3f2 2023-04-18 22:56:27,278 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/1eae5d3a839c45ca88a6cbe3045cba0a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/1eae5d3a839c45ca88a6cbe3045cba0a 2023-04-18 22:56:27,279 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/b1fde4b3f0f545fe920d558efaf0e310 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/b1fde4b3f0f545fe920d558efaf0e310 2023-04-18 22:56:27,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/dcbaafbb6e1e4d93b3215b41a32578b3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/dcbaafbb6e1e4d93b3215b41a32578b3 2023-04-18 22:56:27,281 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/25bbed8dd3b44cd989a6ec2175c7ddae to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/25bbed8dd3b44cd989a6ec2175c7ddae 2023-04-18 22:56:27,282 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/97a57833622a40d984e0179ebb8e77b9 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/97a57833622a40d984e0179ebb8e77b9 2023-04-18 22:56:27,283 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/5e7cd4da8c7245bbbbc3de4e99076c12 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/5e7cd4da8c7245bbbbc3de4e99076c12 2023-04-18 22:56:27,284 DEBUG [StoreCloser-TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/4051a18a53ce4581bf11030e33ed9c6f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/4051a18a53ce4581bf11030e33ed9c6f 2023-04-18 22:56:27,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/recovered.edits/739.seqid, newMaxSeqId=739, maxSeqId=4 2023-04-18 22:56:27,290 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78. 2023-04-18 22:56:27,290 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8181c42622934944fa97cf06c7004a78: 2023-04-18 22:56:27,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:27,292 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=8181c42622934944fa97cf06c7004a78, regionState=CLOSED 2023-04-18 22:56:27,292 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681858587292"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681858587292"}]},"ts":"1681858587292"} 2023-04-18 22:56:27,294 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-18 22:56:27,294 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 8181c42622934944fa97cf06c7004a78, server=jenkins-hbase4.apache.org,35971,1681858424869 in 1.5520 sec 2023-04-18 22:56:27,295 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-18 22:56:27,295 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8181c42622934944fa97cf06c7004a78, UNASSIGN in 1.5560 sec 2023-04-18 22:56:27,295 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681858587295"}]},"ts":"1681858587295"} 2023-04-18 22:56:27,296 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-18 22:56:27,297 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-18 22:56:27,299 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5660 sec 2023-04-18 22:56:27,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-18 22:56:27,839 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-18 22:56:27,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-18 22:56:27,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:56:27,842 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:56:27,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-18 22:56:27,842 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:56:27,847 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:27,849 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C, FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/recovered.edits] 2023-04-18 22:56:27,851 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/00939b41c5cd4892b47110e791fd92b6 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/00939b41c5cd4892b47110e791fd92b6 2023-04-18 22:56:27,852 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/0841f53a2baa4b33b6bd9212a7774c94 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/0841f53a2baa4b33b6bd9212a7774c94 2023-04-18 22:56:27,853 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/104b8918dff94c3093e8576d0c896b7e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/104b8918dff94c3093e8576d0c896b7e 2023-04-18 22:56:27,854 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4596264e40fb4f3c84ec9c058f84c47e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/4596264e40fb4f3c84ec9c058f84c47e 2023-04-18 22:56:27,855 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/f669cfd8747242b39ec927ad03309d6d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/A/f669cfd8747242b39ec927ad03309d6d 2023-04-18 22:56:27,858 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/10f31c27294e4561a71b9285a2321a40 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/10f31c27294e4561a71b9285a2321a40 2023-04-18 22:56:27,859 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/7797ef858a59487aab106868437bea03 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/7797ef858a59487aab106868437bea03 2023-04-18 22:56:27,860 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/afa38f257e0844afa03410a93f276dc7 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/afa38f257e0844afa03410a93f276dc7 2023-04-18 22:56:27,861 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/bc5db5f4e49145f599b2755626b949ce to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/bc5db5f4e49145f599b2755626b949ce 2023-04-18 22:56:27,862 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/c8c8d9b498a7470ea7e6b55295f2051c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/B/c8c8d9b498a7470ea7e6b55295f2051c 2023-04-18 22:56:27,864 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/233d045851904ab0a366d262ab6f0fd8 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/233d045851904ab0a366d262ab6f0fd8 2023-04-18 22:56:27,865 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/26daface7a404826b2efdd06bb85aef3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/26daface7a404826b2efdd06bb85aef3 2023-04-18 22:56:27,866 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/a340438a2006409eb0c56d5afd337f13 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/a340438a2006409eb0c56d5afd337f13 2023-04-18 22:56:27,867 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/e5a71da7015241daaa11c03dc39b557d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/e5a71da7015241daaa11c03dc39b557d 2023-04-18 22:56:27,868 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/fe8d6bb133694e64ae8d62ccf77f3f8c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/C/fe8d6bb133694e64ae8d62ccf77f3f8c 2023-04-18 22:56:27,870 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/recovered.edits/739.seqid to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78/recovered.edits/739.seqid 2023-04-18 22:56:27,870 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/.tmp/data/default/TestAcidGuarantees/8181c42622934944fa97cf06c7004a78 2023-04-18 22:56:27,870 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-18 22:56:27,871 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-18 22:56:27,871 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-18 22:56:27,874 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304180c4f2cecd9c84342a78c52f1e21818ef to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304180c4f2cecd9c84342a78c52f1e21818ef 2023-04-18 22:56:27,875 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418195ff78e320949d49f9c4bcff81b3a59 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418195ff78e320949d49f9c4bcff81b3a59 2023-04-18 22:56:27,876 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304182014e477149c4fe198490f2b6eaef925 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304182014e477149c4fe198490f2b6eaef925 2023-04-18 22:56:27,877 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041820f46ad84fbe445f9ea833bbba396544 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041820f46ad84fbe445f9ea833bbba396544 2023-04-18 22:56:27,878 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041827e103b9221b42828ca636d22cf38a7b to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041827e103b9221b42828ca636d22cf38a7b 2023-04-18 22:56:27,878 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304182889c453ac6e4cff96fd8fa3643d2959 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304182889c453ac6e4cff96fd8fa3643d2959 2023-04-18 22:56:27,879 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041834afa4f600194dedb34b5e680bf1831e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041834afa4f600194dedb34b5e680bf1831e 2023-04-18 22:56:27,880 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041834e6ef0b752b47549dac402f4a9ef87d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041834e6ef0b752b47549dac402f4a9ef87d 2023-04-18 22:56:27,881 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304183c8dcb315caf42bdbe3b4d9c3ceb059e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304183c8dcb315caf42bdbe3b4d9c3ceb059e 2023-04-18 22:56:27,882 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041841853b27e7a24581b77e6361d07531a9 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041841853b27e7a24581b77e6361d07531a9 2023-04-18 22:56:27,883 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041852a6ecdedd5a45ac9f7ceb3cd63ca6b1 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041852a6ecdedd5a45ac9f7ceb3cd63ca6b1 2023-04-18 22:56:27,883 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041859cae9e13ca647cba88b0060164ceeb6 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041859cae9e13ca647cba88b0060164ceeb6 2023-04-18 22:56:27,884 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304185c0e5fd3d8b54e5aa1b08e25929731e3 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304185c0e5fd3d8b54e5aa1b08e25929731e3 2023-04-18 22:56:27,885 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418644a5d04ce9840d2ab25f6df441d04fa to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418644a5d04ce9840d2ab25f6df441d04fa 2023-04-18 22:56:27,886 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041865c0da05276b42419cd6286147f2ae6e to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041865c0da05276b42419cd6286147f2ae6e 2023-04-18 22:56:27,887 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186a3eb57e5cfd4c8db46431aa88ee848a to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186a3eb57e5cfd4c8db46431aa88ee848a 2023-04-18 22:56:27,888 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186fe1f5b9ca5942989171bb65493de8c6 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304186fe1f5b9ca5942989171bb65493de8c6 2023-04-18 22:56:27,888 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187ec1633d56d64437a62b858726772a1d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187ec1633d56d64437a62b858726772a1d 2023-04-18 22:56:27,889 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187fceea7256cd4a46a8810c6cfc8f004c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304187fceea7256cd4a46a8810c6cfc8f004c 2023-04-18 22:56:27,890 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041889c78ffffb34445997fee42d545e37e4 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041889c78ffffb34445997fee42d545e37e4 2023-04-18 22:56:27,891 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041896d17c3f159941c891f221ae6d7b7116 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041896d17c3f159941c891f221ae6d7b7116 2023-04-18 22:56:27,892 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304189e7c96ddbcc9450b9121a9dd9d5c843f to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304189e7c96ddbcc9450b9121a9dd9d5c843f 2023-04-18 22:56:27,893 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418cd795add7d544e7484c699fb0de0683d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418cd795add7d544e7484c699fb0de0683d 2023-04-18 22:56:27,893 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418dafefc8be5014aa4b429b068c741769c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418dafefc8be5014aa4b429b068c741769c 2023-04-18 22:56:27,894 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e09e56dc2178498d91b6d1f7ddbb611d to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418e09e56dc2178498d91b6d1f7ddbb611d 2023-04-18 22:56:27,895 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f68db1e3c2d24c49ba3ba788afbdb547 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f68db1e3c2d24c49ba3ba788afbdb547 2023-04-18 22:56:27,896 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f943a707609c47a9a460d88d0ea8ef7c to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f943a707609c47a9a460d88d0ea8ef7c 2023-04-18 22:56:27,897 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f9ea491021014147bb703ce61f4aa861 to hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230418f9ea491021014147bb703ce61f4aa861 2023-04-18 22:56:27,897 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-18 22:56:27,899 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:56:27,901 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-18 22:56:27,902 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-18 22:56:27,903 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:56:27,903 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-18 22:56:27,903 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681858587903"}]},"ts":"9223372036854775807"} 2023-04-18 22:56:27,904 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-18 22:56:27,905 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 8181c42622934944fa97cf06c7004a78, NAME => 'TestAcidGuarantees,,1681858560248.8181c42622934944fa97cf06c7004a78.', STARTKEY => '', ENDKEY => ''}] 2023-04-18 22:56:27,905 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-18 22:56:27,905 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681858587905"}]},"ts":"9223372036854775807"} 2023-04-18 22:56:27,906 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-18 22:56:27,908 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-18 22:56:27,909 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 68 msec 2023-04-18 22:56:27,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46739] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-18 22:56:27,944 INFO [Listener at localhost/36651] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-18 22:56:27,955 INFO [Listener at localhost/36651] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 229), OpenFileDescriptor=575 (was 575), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=391 (was 383) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=6719 (was 6605) - AvailableMemoryMB LEAK? - 2023-04-18 22:56:27,955 INFO [Listener at localhost/36651] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-18 22:56:27,955 INFO [Listener at localhost/36651] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-18 22:56:27,956 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51f71277 to 127.0.0.1:50825 2023-04-18 22:56:27,956 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:56:27,956 DEBUG [Listener at localhost/36651] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-18 22:56:27,956 DEBUG [Listener at localhost/36651] util.JVMClusterUtil(257): Found active master hash=1728939624, stopped=false 2023-04-18 22:56:27,956 INFO [Listener at localhost/36651] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,46739,1681858423665 2023-04-18 22:56:27,958 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-18 22:56:27,958 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-18 22:56:27,958 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 22:56:27,958 INFO [Listener at localhost/36651] procedure2.ProcedureExecutor(629): Stopping 2023-04-18 22:56:27,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-18 22:56:27,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-18 22:56:27,959 DEBUG [Listener at localhost/36651] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1414fc6e to 127.0.0.1:50825 2023-04-18 22:56:27,959 DEBUG [Listener at localhost/36651] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:56:27,959 INFO [Listener at localhost/36651] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,35971,1681858424869' ***** 2023-04-18 22:56:27,960 INFO [Listener at localhost/36651] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-18 22:56:27,960 INFO [RS:0;jenkins-hbase4:35971] regionserver.HeapMemoryManager(220): Stopping 2023-04-18 22:56:27,960 INFO [RS:0;jenkins-hbase4:35971] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-18 22:56:27,960 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-18 22:56:27,960 INFO [RS:0;jenkins-hbase4:35971] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-18 22:56:27,960 INFO [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(3304): Received CLOSE for f5ed6cadc5eb44dbe5342fc8a2527565 2023-04-18 22:56:27,961 INFO [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:27,961 DEBUG [RS:0;jenkins-hbase4:35971] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ed325d2 to 127.0.0.1:50825 2023-04-18 22:56:27,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f5ed6cadc5eb44dbe5342fc8a2527565, disabling compactions & flushes 2023-04-18 22:56:27,961 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-18 22:56:27,961 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565. 2023-04-18 22:56:27,962 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565. 2023-04-18 22:56:27,961 DEBUG [RS:0;jenkins-hbase4:35971] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:56:27,962 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565. after waiting 0 ms 2023-04-18 22:56:27,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565. 2023-04-18 22:56:27,963 INFO [RS:0;jenkins-hbase4:35971] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-18 22:56:27,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing f5ed6cadc5eb44dbe5342fc8a2527565 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-18 22:56:27,963 INFO [RS:0;jenkins-hbase4:35971] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-18 22:56:27,963 INFO [RS:0;jenkins-hbase4:35971] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-18 22:56:27,963 INFO [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-18 22:56:27,963 INFO [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-18 22:56:27,963 DEBUG [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1479): Online Regions={f5ed6cadc5eb44dbe5342fc8a2527565=hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565., 1588230740=hbase:meta,,1.1588230740} 2023-04-18 22:56:27,964 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-18 22:56:27,964 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-18 22:56:27,964 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-18 22:56:27,964 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-18 22:56:27,964 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-18 22:56:27,964 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-18 22:56:27,965 DEBUG [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1505): Waiting on 1588230740, f5ed6cadc5eb44dbe5342fc8a2527565 2023-04-18 22:56:28,165 DEBUG [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1505): Waiting on 1588230740, f5ed6cadc5eb44dbe5342fc8a2527565 2023-04-18 22:56:28,211 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-18 22:56:28,211 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-18 22:56:28,366 DEBUG [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1505): Waiting on 1588230740, f5ed6cadc5eb44dbe5342fc8a2527565 2023-04-18 22:56:28,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/namespace/f5ed6cadc5eb44dbe5342fc8a2527565/.tmp/info/323991764dc341c7a0d43017f95ab990 2023-04-18 22:56:28,375 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/namespace/f5ed6cadc5eb44dbe5342fc8a2527565/.tmp/info/323991764dc341c7a0d43017f95ab990 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/namespace/f5ed6cadc5eb44dbe5342fc8a2527565/info/323991764dc341c7a0d43017f95ab990 2023-04-18 22:56:28,377 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/.tmp/info/9bb719da84694b7e90f2f7d61de9263a 2023-04-18 22:56:28,379 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/namespace/f5ed6cadc5eb44dbe5342fc8a2527565/info/323991764dc341c7a0d43017f95ab990, entries=2, sequenceid=6, filesize=4.8 K 2023-04-18 22:56:28,380 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for f5ed6cadc5eb44dbe5342fc8a2527565 in 417ms, sequenceid=6, compaction requested=false 2023-04-18 22:56:28,382 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9bb719da84694b7e90f2f7d61de9263a 2023-04-18 22:56:28,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/namespace/f5ed6cadc5eb44dbe5342fc8a2527565/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-18 22:56:28,385 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565. 2023-04-18 22:56:28,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f5ed6cadc5eb44dbe5342fc8a2527565: 2023-04-18 22:56:28,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681858426908.f5ed6cadc5eb44dbe5342fc8a2527565. 2023-04-18 22:56:28,566 DEBUG [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-18 22:56:28,766 DEBUG [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-18 22:56:28,787 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/.tmp/rep_barrier/ea96f8e4b4b34945b60184b7889ba959 2023-04-18 22:56:28,792 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ea96f8e4b4b34945b60184b7889ba959 2023-04-18 22:56:28,966 INFO [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-04-18 22:56:28,967 DEBUG [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-04-18 22:56:28,967 DEBUG [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-18 22:56:29,167 DEBUG [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-18 22:56:29,198 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/.tmp/table/3afc73f0e33b4b07bf405447ca5c53e6 2023-04-18 22:56:29,201 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3afc73f0e33b4b07bf405447ca5c53e6 2023-04-18 22:56:29,202 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/.tmp/info/9bb719da84694b7e90f2f7d61de9263a as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/info/9bb719da84694b7e90f2f7d61de9263a 2023-04-18 22:56:29,205 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9bb719da84694b7e90f2f7d61de9263a 2023-04-18 22:56:29,205 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/info/9bb719da84694b7e90f2f7d61de9263a, entries=22, sequenceid=93, filesize=7.1 K 2023-04-18 22:56:29,206 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/.tmp/rep_barrier/ea96f8e4b4b34945b60184b7889ba959 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/rep_barrier/ea96f8e4b4b34945b60184b7889ba959 2023-04-18 22:56:29,209 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ea96f8e4b4b34945b60184b7889ba959 2023-04-18 22:56:29,209 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/rep_barrier/ea96f8e4b4b34945b60184b7889ba959, entries=6, sequenceid=93, filesize=5.5 K 2023-04-18 22:56:29,210 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/.tmp/table/3afc73f0e33b4b07bf405447ca5c53e6 as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/table/3afc73f0e33b4b07bf405447ca5c53e6 2023-04-18 22:56:29,213 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3afc73f0e33b4b07bf405447ca5c53e6 2023-04-18 22:56:29,213 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/table/3afc73f0e33b4b07bf405447ca5c53e6, entries=9, sequenceid=93, filesize=5.5 K 2023-04-18 22:56:29,213 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 1249ms, sequenceid=93, compaction requested=false 2023-04-18 22:56:29,218 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-18 22:56:29,219 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-18 22:56:29,220 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-18 22:56:29,220 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-18 22:56:29,220 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-18 22:56:29,367 INFO [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,35971,1681858424869; all regions closed. 2023-04-18 22:56:29,375 DEBUG [RS:0;jenkins-hbase4:35971] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/oldWALs 2023-04-18 22:56:29,375 INFO [RS:0;jenkins-hbase4:35971] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35971%2C1681858424869.meta:.meta(num 1681858426589) 2023-04-18 22:56:29,379 DEBUG [RS:0;jenkins-hbase4:35971] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/oldWALs 2023-04-18 22:56:29,379 INFO [RS:0;jenkins-hbase4:35971] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35971%2C1681858424869:(num 1681858426370) 2023-04-18 22:56:29,379 DEBUG [RS:0;jenkins-hbase4:35971] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:56:29,379 INFO [RS:0;jenkins-hbase4:35971] regionserver.LeaseManager(133): Closed leases 2023-04-18 22:56:29,380 INFO [RS:0;jenkins-hbase4:35971] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-18 22:56:29,380 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-18 22:56:29,381 INFO [RS:0;jenkins-hbase4:35971] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35971 2023-04-18 22:56:29,386 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35971,1681858424869 2023-04-18 22:56:29,387 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-18 22:56:29,387 ERROR [Listener at localhost/36651-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@50c61bf7 rejected from java.util.concurrent.ThreadPoolExecutor@54f98537[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2343] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-18 22:56:29,387 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-18 22:56:29,387 ERROR [Listener at localhost/36651-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@40d9a662 rejected from java.util.concurrent.ThreadPoolExecutor@54f98537[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2343] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-18 22:56:29,388 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,35971,1681858424869] 2023-04-18 22:56:29,389 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,35971,1681858424869; numProcessing=1 2023-04-18 22:56:29,390 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,35971,1681858424869 already deleted, retry=false 2023-04-18 22:56:29,390 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,35971,1681858424869 expired; onlineServers=0 2023-04-18 22:56:29,390 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46739,1681858423665' ***** 2023-04-18 22:56:29,390 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-18 22:56:29,390 DEBUG [M:0;jenkins-hbase4:46739] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@26308c53, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-18 22:56:29,390 INFO [M:0;jenkins-hbase4:46739] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46739,1681858423665 2023-04-18 22:56:29,390 INFO [M:0;jenkins-hbase4:46739] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46739,1681858423665; all regions closed. 2023-04-18 22:56:29,390 DEBUG [M:0;jenkins-hbase4:46739] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-18 22:56:29,390 DEBUG [M:0;jenkins-hbase4:46739] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-18 22:56:29,391 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-18 22:56:29,391 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681858426055] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681858426055,5,FailOnTimeoutGroup] 2023-04-18 22:56:29,391 DEBUG [M:0;jenkins-hbase4:46739] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-18 22:56:29,391 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681858426057] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681858426057,5,FailOnTimeoutGroup] 2023-04-18 22:56:29,391 INFO [M:0;jenkins-hbase4:46739] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-18 22:56:29,391 INFO [M:0;jenkins-hbase4:46739] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-18 22:56:29,391 INFO [M:0;jenkins-hbase4:46739] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-18 22:56:29,392 DEBUG [M:0;jenkins-hbase4:46739] master.HMaster(1512): Stopping service threads 2023-04-18 22:56:29,392 INFO [M:0;jenkins-hbase4:46739] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-18 22:56:29,392 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-18 22:56:29,392 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-18 22:56:29,392 ERROR [M:0;jenkins-hbase4:46739] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT 2023-04-18 22:56:29,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (1250602145) connection to localhost/127.0.0.1:35337 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-18 22:56:29,393 INFO [M:0;jenkins-hbase4:46739] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-18 22:56:29,393 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-18 22:56:29,393 DEBUG [M:0;jenkins-hbase4:46739] zookeeper.ZKUtil(398): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-18 22:56:29,394 WARN [M:0;jenkins-hbase4:46739] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-18 22:56:29,394 INFO [M:0;jenkins-hbase4:46739] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-18 22:56:29,394 INFO [M:0;jenkins-hbase4:46739] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-18 22:56:29,394 DEBUG [M:0;jenkins-hbase4:46739] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-18 22:56:29,394 INFO [M:0;jenkins-hbase4:46739] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-18 22:56:29,394 DEBUG [M:0;jenkins-hbase4:46739] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-18 22:56:29,395 DEBUG [M:0;jenkins-hbase4:46739] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-18 22:56:29,395 DEBUG [M:0;jenkins-hbase4:46739] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-18 22:56:29,395 INFO [M:0;jenkins-hbase4:46739] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.05 KB heapSize=393.72 KB 2023-04-18 22:56:29,408 INFO [M:0;jenkins-hbase4:46739] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.05 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/fedeb4ea56df488ca7fc85a66c94259c 2023-04-18 22:56:29,412 INFO [M:0;jenkins-hbase4:46739] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fedeb4ea56df488ca7fc85a66c94259c 2023-04-18 22:56:29,413 DEBUG [M:0;jenkins-hbase4:46739] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/fedeb4ea56df488ca7fc85a66c94259c as hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/fedeb4ea56df488ca7fc85a66c94259c 2023-04-18 22:56:29,416 INFO [M:0;jenkins-hbase4:46739] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fedeb4ea56df488ca7fc85a66c94259c 2023-04-18 22:56:29,416 INFO [M:0;jenkins-hbase4:46739] regionserver.HStore(1080): Added hdfs://localhost:35337/user/jenkins/test-data/35872bcd-daa6-197d-018e-349c75419ea0/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/fedeb4ea56df488ca7fc85a66c94259c, entries=65, sequenceid=578, filesize=24.7 K 2023-04-18 22:56:29,417 INFO [M:0;jenkins-hbase4:46739] regionserver.HRegion(2948): Finished flush of dataSize ~345.05 KB/353327, heapSize ~393.70 KB/403152, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 22ms, sequenceid=578, compaction requested=false 2023-04-18 22:56:29,418 INFO [M:0;jenkins-hbase4:46739] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-18 22:56:29,418 DEBUG [M:0;jenkins-hbase4:46739] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-18 22:56:29,421 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-18 22:56:29,421 INFO [M:0;jenkins-hbase4:46739] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-18 22:56:29,422 INFO [M:0;jenkins-hbase4:46739] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46739 2023-04-18 22:56:29,424 DEBUG [M:0;jenkins-hbase4:46739] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,46739,1681858423665 already deleted, retry=false 2023-04-18 22:56:29,489 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-18 22:56:29,489 INFO [RS:0;jenkins-hbase4:35971] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,35971,1681858424869; zookeeper connection closed. 2023-04-18 22:56:29,489 ERROR [Listener at localhost/36651-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@38e8ed23 rejected from java.util.concurrent.ThreadPoolExecutor@54f98537[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2343] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-18 22:56:29,490 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4c1da7] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4c1da7 2023-04-18 22:56:29,490 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): regionserver:35971-0x101997ad1e40001, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-18 22:56:29,490 ERROR [Listener at localhost/36651-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7c33bc6d rejected from java.util.concurrent.ThreadPoolExecutor@54f98537[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2343] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-18 22:56:29,491 INFO [Listener at localhost/36651] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-18 22:56:29,589 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-18 22:56:29,589 INFO [M:0;jenkins-hbase4:46739] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46739,1681858423665; zookeeper connection closed. 2023-04-18 22:56:29,589 ERROR [Listener at localhost/36651-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@715d92ab rejected from java.util.concurrent.ThreadPoolExecutor@5f408bed[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3917] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-18 22:56:29,590 DEBUG [Listener at localhost/36651-EventThread] zookeeper.ZKWatcher(600): master:46739-0x101997ad1e40000, quorum=127.0.0.1:50825, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-18 22:56:29,590 ERROR [Listener at localhost/36651-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2d5fb51 rejected from java.util.concurrent.ThreadPoolExecutor@5f408bed[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3917] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-18 22:56:29,592 WARN [Listener at localhost/36651] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-18 22:56:29,597 INFO [Listener at localhost/36651] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-18 22:56:29,705 WARN [BP-835922315-172.31.14.131-1681858420082 heartbeating to localhost/127.0.0.1:35337] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-18 22:56:29,706 WARN [BP-835922315-172.31.14.131-1681858420082 heartbeating to localhost/127.0.0.1:35337] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-835922315-172.31.14.131-1681858420082 (Datanode Uuid 0a64437a-398a-4328-a5cc-cc24e037595c) service to localhost/127.0.0.1:35337 2023-04-18 22:56:29,708 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/cluster_e7f6bc09-fd33-5a3b-ac62-21cde829d37b/dfs/data/data1/current/BP-835922315-172.31.14.131-1681858420082] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-18 22:56:29,709 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d68229e7-e7b9-bc35-f09c-8b8d6df15827/cluster_e7f6bc09-fd33-5a3b-ac62-21cde829d37b/dfs/data/data2/current/BP-835922315-172.31.14.131-1681858420082] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-18 22:56:29,746 INFO [Listener at localhost/36651] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-18 22:56:29,920 INFO [Listener at localhost/36651] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-18 22:56:30,083 INFO [Listener at localhost/36651] hbase.HBaseTestingUtility(1293): Minicluster is down